On the use of memory

The difference in memory in computers and organic memory is a strange thing. The memory in computers is linearly aligned and stores elements very differently than organic memory. Where the storage area is not large enough in computers, we tend to use databases to make the memory "searchable" and mallable.

Computers generally need to trawl through the entire space in order to find something, albeit with clever algorithms rather than an exhaustive search. We're modeling the world around us into a specific, definition to work with and then compare new input to those things we know.

Organic memory is very different. Through the input of elements and situations, the recall of other elements in memory seems activated automatically without the need for a search. As if very small biological elements recognize features in the stream of information and respond. This may cause other cells to become activated too, thereby leading to the recognition of one set of features with previous experiences. It's as if the recognition drifts up automatically, rather than that a general CPU searches for some meaning based on a description or broken down representation.

The problems of computers then is our limitation / lack of imagination to be able to represent memory in a non-linear form. The way how computers are modeled require the programmer to define concepts, elements and situations explicitly or less explicitly using rules and then search, within a certain space, for similar situations and reason further from there. It's totally lost if the situation cannot be mapped to anything.

I was thinking lately that it would be a ground-breaking discovery if memory could be modeled in similar ways to organic memory. That is, the access to memory being non-linear and a type of network, rather than developing a linear access algorithm. If you consider a certain memory space where a network may position its recognition elements (neurons?), then the connection algorithm of a current computer should map the linear memory differently into a more diffuse space of an inter-connected network. Basically, I'm not saying anything different than "create a neural network" at this time, but I'm considering other possibilities to use the mechanical properties of memory access in a clever way and as such to reduce the required memory for connection storage and to figure out a possibility to indicate "similarity" by the proximity of memory address locations.

Or, alternatively to that, use a neural network to determine a set of index vectors that will map to a large linear space. The index vectors can be compared to a semantic signature of any element. This signature should be developed in such a way that it is categorizing the element from various perspectives. Basically, considering the ability for semantic indexing of text, the technique is used to find texts that are semantically similar.

The larger the neural network, the finer its ability to recognize features. But our minds do not allocate 100 billion neurons to the (same) ability of pattern recognition. Thus, you could talk of specialized sub-networks of analysis that together define a certain result (binding problem).

But perhaps we're thinking too much again in the terms of input-processing-output as I've indicated before. We like things to be explicitly defined, since it provides a method of understanding. What if the networks don't work together in a hierarchy (input->network1->network2->output->reasoning), but work together in a network themselves?

Then this would mean that such a network could aggregate information from different sub-networks together to form a complicated mesh itself. The activation of certain elements in one part could induce the activation of neurons in another, leading to a new sequence of activation by reasoning over very complicated inputs of other neuronal networks. For example, what if thinking about a bear causes our vision analysis network to fire up / become induced and then produce a picture of such a bear?

Imagine a core of a couple of neural networks that have complicated information available to them from "processing" neural networks before them. If those core networks are interconnected in intricate ways and influence one another, then it's likely that a smell causes the memory of a vision or hearing in another network, albeit slightly weaker than normal.

Leaving this thought alone for now...

Reusable A.I.

A miscellaneous crazy thought I had today was to look into the potential of reuse for artificial intelligence algorithms or networks. Think sharing the factors, definitions, neurons... Persisted artificial neural networks, which is only possible in computers because they are determined and extractable from memory.

The different functions of sight, smell, hearing and so forth are always seen as very specific properties of a system and it's specialized after that. What would happen once the networks are interchangeable, pluggable and can start talking to one another? Even then we'll need to make decisions on which networks to support and which not to, but next to this problem, the underlying platform... the glue between the networks, could serve as a basis for connecting grid / cloud computing of A.I. networks that each serve a particular purpose.

Some very complicated problems require some heavy processing power. I strongly believe that subdividing the problem space into smaller problems, each with its own expertise and domain, can help to significantly decrease the eventual complexity and required hardware.

Bayes theorem

In artificial intelligence courses, a start into an exploration of the Bayes theorem. In simpler words, Bayes is about discovering (strengths of) relationships between cause and effect. It's looking at events and developing a hypothesis or observing an occurrence which may have led to that event and an expression of the chance that the event occurs based on the occurrence/trueness of the hypothesis (potential cause).

This theorem is used in medical analysis for example (what is the chance for a person to have meningitis, considering the person has a headache?). Of course, the theorem can be expanded by the union of two hypothesis's. When both occur at the same time, *then* what is the chance for the event (meningitis) to occur?

For another direct application, consider credit-card fraud. If you have a large training set and you have a large database of credit-card frauders, you could determine from the data-set the probability that a female, a certain age group or people from a certain neighborhood/city/area commits credit card fraud. You could theoretically even come up with a number for a credit card applicant to state the probability the person would commit credit card fraud and so on. Of course, maintaining the view that you're dealing with probability, not certainty.

Bayes can also be used by learning systems to develop associations or relationships. It doesn't thus produce a boolean true|false relationship between elements, but a probability relationship. Theoretically, I think it should even be possible to organize different kinds of relationships (dependencies, associations, classifications) using Bayes just by looking at a large data-set. The problem here is that such an engine shouldn't be looking at all possible combinations of cause / effect, but logically reason within those, so make deductions about possibly sensible combinations.

Then one can question whether we as human beings absolutely exclude some silly statements. If we did employ true|false for each hypothesis with nothing inbetween, then we would have trouble understanding the world around us too, since it's full of exceptions. Does this suggest that some sort of Bayesian theorem is at the basis of our association determinations in a neural network?

http://www.inference.phy.cam.ac.uk/mackay/Bayes_FAQ.html

So, it's interesting to read about this...

Semantic priming

Psychology and neuro-science has done research on the effects of semantic priming. Priming is the effect where you're prepared to interpret a certain word, vision or thing in a certain way. When you're primed to react to a soon-to-be-fired stimulus and you're paying attention to it and the stimulus actually occurs, the reaction time is very quick. As soon as another stimulus is given however, the reaction to that is generally very slow.

I think priming is very interesting in the context of semantic processing of text or the world around us, in that a context of a story may also prime us for what comes next. That is, we always build up some kind of expectation of what will happen next and read on happily to see what's really going to occur.

Some time ago, I posted something about latent semantic indexing. It's more or less an indexation of latent semantics. Latent means:

1.	present but not visible, apparent, or actualized; existing as potential: latent ability.

2.	Pathology. (of an infectious agent or disease) remaining in an inactive or hidden phase; dormant.

3.	Psychology. existing in unconscious or dormant form but potentially able to achieve expression: a latent emotion.

4.	Botany. (of buds that are not externally manifest) dormant or undeveloped.

So, latent means hidden or dormant. It's about semantic meaning that you can't see inside the text, but use as a key for indexing that meaning or definition. In other posts, I doubted the viability of constructing formal knowledge systems (where knowledge is explicitly documented), due to the huge size and the huge efforts required in defining this knowledge (and the obvious ambiguities and disagreements that go along with it). Other than that, knowledge is also dynamic and changing, not static.

Considering priming thus and binding this with a technique for latent indexing, one could achieve a system where related symbols are primed before they are interpreted. Given different indices for vision, smell, audio and somatosensory information, each specific index could eventually (without saying how) be made to point to the same symbol, thus strengthening the interpretation of the world around a robot or something similar.

Thus, rather than explicitly defining relationships between concepts, consider the possibility of the definition (and growing) of indexed terms which partially trigger related terms (prime the interpreter), as the interpreter moves on to the next data in the stream. This could allow a system to follow a certain context and distinguish relationships of things in different contexts, because the different contexts have different activation profiles of each symbol.

Coupling this with probability algorithms, it would be interesting to see what we find. In fact, using probability is the same as the development of a hypothesis or "what-if" scenario. Whereas a certain relationship does not yet exist, we seek ways to prove the relationship exists by collecting evidence for it.

Some other activities that we learn are subconsciously learned. That is, the action/reaction consequences of throwing an object and having it drop on the floor. If the object is of metal, it probably won't break. If it's made of glass, it'd probably shatter. Those things are not obvious to children, but can quickly be learnt. Glass is transparent, feels a certain way, and there are a number of standard elements which are generally of glass. Plastic looks similar, but makes a different sound. We should aim to prevent dropping the glass on a hard floor. This bit of knowledge is actually a host of different relationships of actions, reactions, properties of objects either visible or audible and by combining these things together, we can reason about a certain outcome.

The psychology book also importantly notes the idea of attention. It specifically states that when attention is not given, performance of analysis, reasoning or control drops significantly. This means that we're able to do only one or two things at a time. One consciously, the other not so. But that it's the entire mind with control, audible and visible verification mechanisms to control the outcome.

The interesting part of this post is that it assumes that symbols as we know them are not named explicitly by natural language, but are somehow coded using an index, which has been organized in such a way that neighboring indexed items become somewhat activated (primed) as well to allow for the resolution of ambiguities. An ambiguity is basically the resolution of two paths of meaning, where the resolution should come by interpreting further input or requesting input from an external source in an attempt to solve it (unless assumptions are made to what it means).

Another thing that drew my attention is that recent strongly primed symbols may be primed strongly in the future independent of its context. This is mostly related to audio signals and related to for example the mentioning of your name. You could be in a pub hearing a buzz, but when your name is called somewhere, you can recognize it immediately within that buzz (thus, the neurons involved in auditory recognition are primed to react to it).

It's probably worthy to extend this theory by developing the model further and considering human actions, reasoning, learning and perception within that model (as opposed to building a network and trying out how it performs). Since it's already very difficult to re-create human abilities using the exact same replicas of biological cells, why not consider simpler acts and verifying parts of this reasoning with such a smaller network?

The first elements of such a network require a clever way of indexing signals and representations. In this way, the indexing mechanism itself is actually a clever heuristic, which may re-index already known symbols and place it in a different space. The indexing mechanism doesn't feel static.

Pre-processors of information

The psychology course I'm taking requires reading through a pretty large book (albeit in not too small type and with loads of pictures). The sensory system is explained, so it's sometimes more like a biology book. It's basically stating that rods are to analyze dim-lit places and cones are for richer-lit places. The cones can discern color and have lower sensitivity.

Researchers have determined that right after the light is transduced after the cones and rods, that nerve cells already start pre-processing the information. You should compare this pre-processing to the execution of an image filter of Photoshop. It runs some edge detection filters for example, improves contrast here and there and then sends it back to the primary visual cortex for further analysis.

I'm taking some personal experiments by looking at a scene for 1-2 seconds, then closing my eyes and attempting to reconstruct that scene. Looking at it for longer or more frequently makes the image more perfect, but the actual image with eyes open has a lot more information than that which I can reliable reconstruct. Or rather... I can reconstruct details of A tree or road, but possibly not THE tree or road out there. So my belief system of what I think a tree is or a road is starts to interfere. The interesting thing is that the actual image I can reconstruct is mostly based on the edges and swats of colors.

Example detail image

Example "memory" image

It's not very clear (due to time constraints) that the middle part of the picture still has the highest detail, whereas the sides of it has less due to peripheral vision.

The mind thus deconstructs the scene first by edge detection, finding lines, but at the same time highly depends on the ability to identify complete objects. Very small children for example are already surprised or pay attention when objects that were thought to be together suddenly seem to be actually apart.

It does take some time to identify something that we've never seen before, but pretty quick we're able to recognize similar things, although we may not know the expert name for it.

By deconstructing the scene, you could say it also becomes a sort of "3D world" that we can personally manipulate and visualize further (mind's eye). So I don't think we're continuously re-rasterizing heavy and complex objects, but have the ability to consider an object whole by its edges/traces, then rotate it, translate it or do with it as we please.

In these senses, the sciences that deal with signal processing and so on should depend on these techniques heavily. It is possible to recognize objects through its pixels, but perhaps by running filters on it before, the features are easier detected and the pattern recognition mechanism might just be significantly better. Thus... the way in which signals are presented probably always require pre-processors before they are sent to some neural network for further processing. In that sense, the entire body thinks, not just the brain.

A.I. is about search

Well, so far I've enrolled on a couple of courses. One of them being "A.I. Kaleidoscope". It's a great course with very good course material and exceptional course material from the professor.

The book I'm reading, called "Artificial Intelligence", is very well written and highlights a couple of philosophical understandings as well as explains mathematical underpinnings of A.I. that have established so far. So it's quite a broad area it is discussing.

One of the statements I come across is that A.I. is about searching problem spaces. Whereas some problems have algorithms, other problems have a state space, where states are mapped onto and where transitions from one state to another are shown as arcs in a graph. Bringing the discussion to graph theory and trees. And breadth-first and depth-first searches, heuristics, and so on.

The idea is thus that A.I. is about mapping knowledge within a certain domain and understanding the phases or steps that an expert goes through in order to come to a reasonable conclusion (reasonable meaning not necessarily optimal, but certainly acceptable).

In previous posts, I sometimes discussed that we human beings aren't necessarily purely rational, but act emotionally as if we're programmed. We think we're exceptionally clever though. Well, another part of the book discusses the fact that we only consider things intelligent that act in ways that we ourselves would and could do. You could argue for example that intelligence shouldn't be subject to such a "narrow?" definition. But philosophically, there is no common agreement on the actual definition of intelligence, so this discussion isn't that useful at this time (within a blog that is).

I'd like for the moment to disconsider the general "folk" consensus that intelligence is solely determined by human observation and imitation (dolphins are considered intelligent because they seem able to have intricate conversations in their speech and behave in seemingly human ways to our stimulus and interactions). Taking thus a slightly wider interpretation of intelligence, and accepting the statement that "A.I. is about search", you can only conclude that Google built the most intelligent being on the planet. It's capable of searching through 70% of the internet at lightspeeds, you always find what you're looking for (unless it's too specific or not specific enough) and so on. Now, perhaps the implementation isn't necessarily intelligent, but the performance of the system surely demonstrates to me, using my browser, that it's a very intelligent system.

One important branch in A.I. is about "emergence", something I blogged about recently. It's when simple individuals within their own little context and environment execute actions, which within a greater context build up to a very intricate system that no individual could control, but together displays highly sophisticated attributes of intelligence. An example could be free market mechanisms. You could say that the information that a single individual has to control the logistics of vegetables in a single city would be limited, and most likely a single individual couldn't optimize this task. But all vegetable sellers in a certain city are very likely to be apt in optimizing their local inventory in such a way that it has least waste and optimal profit. Optimal profit means having just enough for people in their local environment to benefit, but not too much to have it thrown away.

These "agents" as they are called in A.I. act on their immediate environment. But taken together on a higher level, their individual actions contribute to a higher level of intelligence or optimization than possibly a single instance, computer, individual or thing could be if they were to understand the entire problem space, understand it and optimize in it. The core of the above is that many intricate and complex systems consist of simple agents that behave according to simple rules, but by consistently applying these rules, they can achieve "intelligence" that far exceeds their individual capacity.

So, A.I. does seem to be about search, but it's not about finding the optimal. Maths is about finding optimals and truths, it's an algorithm, thus (must be/needs to be) absolute and consistent. A.I. is about a problem space, possible solutions and trying to find optimal solutions (applying "intelligence") as best as you can, but always taking into account the cost to get there.

Humans don't always find optimal solutions to problems. They deal with problems at hand and are sometimes called "silly" or "stupid" by other humans (agents).

One of the things I liked about the book is that "culture" and "society" are instrumental to intelligence. It clearly suggests that there's a need for interaction for intelligence to occur. In fact, for intelligence to exist. It highly suggests that intelligence is thus cultural, but also infused and created by the culture itself.

If A.I. is about search, and more recent posts are about semantic models, where does this leave neural networks? I think the following:

1. You can't build a human brain into a computer due to memory, bandwidth, cpu and space constraints. So forget about it.
2. A.I. shows that you can model certain realities in different ways. There are known ways to do this through graphs, but those graphs have too harsh and clear relationships between them. They should be softer.
3. Searching a space doesn't exclude the possibility of indexing knowledge.
4. Relational databases may have tables that have multiple indices. Why not knowledge embedded in A.I. systems with multiple entry points, based on the input sensor?

Thus... what if we imagine an A.I. which behaves unlike the human brain but in other ways like it, uses multiple "semantic" indices for interpreting certain contents and contexts?

Latent Semantic Indexing is a technique to describe a certain text and then give it some sort of index (rating)?. You could then do the same to another piece of text and compare the two. The rate to which the two are alike is a certain score for the similarity. Thus, LSI could serve as a demonstration of the technique for semantic indexing (and possibly storage) of other receptors as well (sensors/senses).

Imagine that a computer has access to smells (artificial nose), images (camera), audible sounds (microphone) and so on and it has the ability to maintain a certain stream of this information in memory for a certain amount of time. The information together is a certain description of the current environment. Then, we code the current information using an algorithm yet to be constructed such that it can be indexed. And we create a symbol "A" in the table (the meaning) and create indices for smell, vision and hearing to point to A. Any future perception of either the smell, or the vision or the hearing might point to A, but not as strongly as when all indices point to it (confusion).

The problem space in this example is more limited to the combination of the senses and what it means and searching for possible explanations within each "sense" area.

The difference with more classic A.I. is that the classic version attempts to define context and define reality IN ORDER to classify it. The above version doesn't care much about the actual meaning (how we experience or classify it with our knowledge after x years of life). It cares about how one situation is similar to another one. In that sense, the definition of meaning is about how similar some situation is to another.

Now... if the indices are constructed correctly, similar situations should be close to one another. Thus, a computer should be able to quickly activate other memories and records of possibly similar situations.

VU start

I visited the university today because of the start of colleges. Picked up most books except a few and then... start studying in the evenings and weekends.

The recent post about frozen realities is a nice one to extend further. The meaning and definition of "intelligence" is also one to think about before one calls a system "intelligent". It's used as a buzzword. As wikipedia states it:

"Intelligence (also called intellect) is an umbrella term used to describe a property of the mind that encompasses many related abilities, such as the capacities to reason, to plan, to solve problems, to think abstractly, to comprehend ideas, to use language, and to learn. There are several ways to define intelligence. In some cases, intelligence may include traits such as creativity, personality, character, knowledge, or wisdom. However, most psychologists prefer not to include these traits in the definition of intelligence." (source: wikipedia).

As with "emergence", we should not consider intelligence to be subject to one entity within a system, we should remain open enough to allow definitions where relating entities together, as part of one system. demonstrate intelligent activities. Just as with ants, it's possible with computers that interacting components are smarter than the sum of their individual parts.

The following is a current imagination and I may change my mind on it. Think of the human brain as a set of components that control motor functions and analysis / pattern recognition functions. The ability and function of each component is pre-determined through DNA, but the way how they interact with other components is to be learned. A supervisor in the brain controls through feedback mechanisms of the input sensors whether a certain desired result was achieved. The desired result is also dynamic and a pattern.

Actually, everything is a pattern of some kind. The patterns are stored in huge containers where each container has patterns of the same type. Pattern recognition and indexing sounds like a very complicated affair. For example, the smell of hot pizza is not just "pizza". There is no such smell. It's the smell of cheese, dough, hot cheese, tomato and everything else what's on there. The individual smells make up the rest. Depending how trained you are, we could still wonder whether someone can guess what's in the oven? The more information we use to construct our environment, the more senses we need to make sense of it.

We tend to automatically direct other senses for the confirmation of certain impulses. Such as looking into the oven, listening intently to some events and so forth.

The idea here is then that it sounds difficult for a single sense to function properly by itself. It's missing a lot of information to "get around". So the conjunction of patterns from different senses can be used to provide a deeper definition of the environment than a single sense can.

A very difficult thing is visual recognition and deconstruction. I don't think we're able to actually store every pixel of everything we see. Rather, I believe we store some kind of gist, a simple three-dimensional deconstruction of the image and some color features for each. This could differ from person to person of course and explains why certain people are artists and others are not :). The ability to see perspective is a very difficult one. We're able to see that, because we know that some things are larger than just what we see, thus we know something is in front of something else. Also, we strongly use shadows and tint differences to further analyze a scene.

In OpenGL, we speak of a pipeline to construct a scene. But when you look at an image, we also need to think of a pipeline for de-rasterization. Possibly, as soon as a computer is able to construct simple wireframes from simple images, we're steps closer to creating a computer that can recognize objects easier.

Whereas many people consider the definition of things equal to "naming it", this is also wrong I think. A definition of something can also be considered: "a formal recognition and method of communcation through symbolic means, such that it calls up the same pattern and recognition with another intelligent being". Thus, we need not restrict ourselves to using words. If we were telepathically endowed and could transfer our thoughts, we'd probably call that "definition" instead.

So, calling something "car" then is just a specific name for a specific generic pattern. Moreover, the specific car that one has in mind is likely different from somebody else. So the patterns we really evoke are possibly different from person to person, yet we all catch the gist of the message.

If you consider that a pattern can be associated with a term, why not consider the possibility that certain indexes can be given a certain name or list of names?

Then, intelligence becomes the ability to reason with those patterns that are similar to some extent in order to make something out of it that re-defines your reality. A certain kind of juggling and analysis on similarity, possibilities, abilities and relationships (plus defining new ones, aka learning) where certain rules exist that should not be broken. For example, most cars can't drive on water. When your car leaves, so do its tires. A car should not drive with its door open. A car should have wheels to drive.

The problem of reasoning then is how rules are embedded within this reasoning system. The patterns *are* the symbols and the names it has been given. Semi-patterns that are lightly activated form new possible paths. Is it that the path of one pattern to another forms the embodiment of a rule?

The frozen model of reality

As soon as you've gotten out of the coding trenches as a developer, you'll soon be confronted with the need to model reality. Basically, the idea about software development is to take a snapshot in time of reality that works for the current set of requirements and then automate it into some program or system. The larger the system, the more difficult the freeze gets and thus the more frequent are the change requests or people trying to push things through.

The problem, obviously, in freezing reality is that as soon as you defrost it again, you'll need to find a new equilibrium or frozen point in time, the target. The ability to move from one frozen state to another is your flexibility of the architecture, the clarity of the solution to information analysts and everything and everyone inbetween that can put their foot in the door or wants to have anything to do with it. Successful implementations tend to attract a lot of success-share-wannabees, poor projects tend to attract a lot of attention from people that know how it's done and people that refuse to work that way.

Anyway, the problem with freezing reality and development is that you're in a stop & go process. You can never go in a continuous changing way to some new hybrid form of operation, it's always a little bit or a lot, come to a complete stop, wait, then determine your new direction. Confusing to many, since we tend to differ in opinion on the best direction to take afterwards, or even differ in opinion what the frozen state looks like or even what the soon-to-be-defrosted state should look like.

The freeze is required because development and coding is the formalization of rules and processes in a particular moment in time. Thus, software engineering is basically freezing reality as little as we need to, but as much as we should, to make us more effective from that point onwards. Luckily we still employ people that use software and can think up their own ways around us to still enable a business to grow and bring it forward, otherwise we'd really be in the ... .

Anyway, a very simple conclusion could be that any formalization of reality for a period of time is therefore subject to inflexibility in the same way that a formal representation of anything is just a particular perspective of that thing in time (and fashion?).

If you look into the problems of software engineering, the actual problems that we still encounter nowadays have not changed a single bit, but the technologies have. Any new technology comes with promises that "modeling" the enterprise with that technology is going to make it more flexible, yet it always starts with the formal chunking of reality so that a program can actually carry out some work. It's true that technologies have made it easier and faster to develop programs, mostly the 1GL, 2GL, 3GL and 4GL phase and we're getting closer to the business language due to methods of specification, but we're not changing the real method behind it, the formalization of reality at a point in time.

In order to make machines really intelligent, we should exceed our own limitations, since we depend on formalized and static models to comprehend something and from those models we re-build our solutions.

As an example, I imagine an artificial intelligent system that doesn't attempt to formally describe an object once, but reshapes and details the object as soon as more information becomes available to describe it and should probably even be able to split objects into different ones as soon as a major axis of separation (a category) becomes available to make it distinct.

Depending on who you ask in life, people give you different answers on trees. Some people know only one tree: "the tree". Other people know their pine trees from their oak trees and yet other people can identify trees by their leaf silhouette. So somewhere and somehow, we tend to further categorize items as soon as we get swamped by too many symbols in the same category. We're luckily very apt in finding specific differences between types and especially how they are common, so that the categories have valid descriptors.

But... one does not grow up and think of a tree as a pine or an oak, we think of it as a tree first, then later it is identified as a tree of a specific type. We can use smell and vision to identify a pine tree, even tactile functions. The combination of smell and vision is a very powerful identifying function, vision alone or smell alone might still throw us off.

Now, making this post a bit specific to python. Python has a process called "pickling" that is used to persist objects in storage space. In artificial intelligence, the neural network often acts in certain phases. The phase were it learns and adjusts according to feedback and a phase where it executes and recognizes new input symbols. We're too afraid to let the network run in those two modes at once. Either because we can't predict the result or because we're not using the right model for it even then. The human brain though is constantly stimulated by recognizing something it saw before that was alike, but slightly different or still "exactly the same". I believe we're constantly adjusting our images, smells and other input signal patterns as we experience them.

But without a suitable learning process on the outside that is connected to a working learning machine in such intelligence, it won't go far. In this view, I'm considering that the human brain has the innate capacity to learn about symbols, but just needs to experience the symbols to make sense of them eventually how they interact and relate to one another. It's not very uncommon to meet somebody that has entirely different viewpoints and experiences, or interpretation about their environment than you do.

Thus, the problem in A.I. at this time considered isn't necessarily so much about how we define a proper ontology (since that ontology is also based on the current snapshot and perspective we have, our "frozen model"), it's about how we define a machine that knows nothing about symbols, but has the capacity to understand how symbols (patterns from different inputs) relate together and perhaps even has the ability to reason with those symbols, but that's taking it another step further.

I'd recommend that after this post, you keep track how much detail you're observing in your everyday world in different situations. You'll be amazed how much you 'abstract away' from the environment and even though you see it doesn't mean that you notice it. And it's also possible that missing things create strange situations in which you notice that it's not there, even though you expected it to be. Or without it, it doesn't exactly look like the object as you expected. That change in observation, should and does that change your reality? Does it change how you observe the world in the future? Is that the real process of learning? Continuous adaptation of reality based on observation?

AI ambience or collective web intelligence?

I've been busy a lot with some administrative things. In september classes are starting and I'm commencing with A.I. Registrations are done, just register for a couple of courses and go. There's a new direction for the next year: "Human Ambience". I had quite some interest into intelligent systems, but I do like ambient intelligence as well. I think it's really done well when you don't notice it at first, but then later go: "oh, that was actually pretty cool!".

For the rest Project Dune is trucking on as usual. I'm preparing a plan for a manager of the company I work for to possibly spend a bit of budget on getting the project a little bit further, but then with the help of the effort of some colleagues. All done open source of course. So that's exciting.

I'm also looking at perhaps providing a Java API to interface with CUDA. The objective is to make it available to Java users. Not sure how to write the "java" program and compile that for CUDA use though :).

Communication, interpretation and software engineering

A majority of problems in software engineering are due to inefficient social activities, well... beyond poor estimation, poor assessment and poor verification of course. :)

Here's a very nice website I found:

http://www.radio-subterranean.com/atelier/creative_whack_pack/pack.html

It's focused on creativity, but can be applied to innovation and some of those issues can be applied to general software engineering that's simply the development of applications.

There are still quite a lot of apps out there that have not been designed properly. They start out, but from that point are already dead in the water. It's simply no use to extend it further. It may work when it's done, but every effort spent on it simply isn't worth it. "Design-dead", it's called. This can be prevented if you look for sounding boards and more experienced peers. Don't be proud, let others contribute and seek some help in what you're doing.

The other thing are assumptions or assumptions from business that turn into dreamed up requirements but eventually appear to be problems. Especially for security this could be the case.

The latter problem is mostly the lack of root analysis. Rather than directly assimilating what a person wants, it's about asking the question why something is wanted and what the root problem is. Many, many times, people come to you directly with a request to do something which they consider the resolution to their problem. They're not telling you the problem they're having. Keep on asking, identify why something is wanted and maybe it's possible to come up with a much easier alternative.

Lack of definition is another. It's difficult for some people to understand that others are not experts in the same domain. Spend a bit more time to explain your own process and activities, then see if you can develop a correct cross-over between the two domains of expertise for an optimal result.

Well, and further... It's mostly about continuous communication and verification. Going off for half a year and then coming back with an end result is bound to give a lot of deviations from ideas. Maybe the ideas were wrong in the first place, maybe the interpretation. It doesn't matter at that point.

Software Architecture "methodology"

As a software architect, buzzwords and the buzz itself comes at you in various guises. Sometimes it's the business itself that heard about company X employing methodology Y and they want to do the same. Most of the times it's a hotshot from company Z that wants to change the world and do things radically different, promising totally different flexibility and capabilities.

I'm a bit tired of all these words. The only thing that counts are activities, as they produce the output that can really be used, and revolving around that output are only a number of factors. No single methodology will ever succeed in setting up a dynamic company. It's the people themselves, how they interact and communicate, the level of politics, the culture and how quickly the development team picks up new technologies that are really worthwhile.

Architecture methodologies are basically all about the same thing... guidelines about how to execute the development activities and a registration of the foreseen limitations and recommendations for that team.

The latest thing is of course SOA. It's often thought of as a new methodology, but it's actually just a different perspective on your organization. It cannot be called an architecture though. It does force systems and data to be organized a certain way, but it does not dictate how. So, SOA is a direction that a company decides to take, which might make sense, but it isn't anything else than that.

I see EAI architectures at the core of architecture, influenced by SOA, whereas SOA is the organization of the infrastructure into a set of services. It's as if SOA is basically a aggregation of resources to make it an artificial and digital interface that other organizations can use. So, SOA is more like a vertical whereas EAI definitions are mostly describing the trees of the "what-otherwise-would-be" forest of resources. EAI defines the guidelines in getting the resources set up, SOA defines what to expose. EAI is driven by the output of SOA and SOA is limited by the level of organization in EAI.

Once you think about it, the only thing that changes are the levels of connectivity and the level of digitalization. Organizations in the 60's were already capable of exchanging information, it just mostly traveled on paper. Currently, the velocity and size of information has grown so much that paper processing is just not an option any more. That is why the complexity of security and interpretation is now being born by digital systems, whereas this previously was interpreted and verified by human beings.

So, although SOA makes it really interesting for companies, it's far from a silver bullet, although pulling off a successful infrastructure of that kind will very likely make you more flexible. The problem is however that in every architecture, a number of assumptions are always baked in. Sometimes, when businesses significantly change, the architecture can't change along with it, thereby requiring large investments to bring about that change. Every business and organization has a "heart" to it where data is kept or basic business processing is done. If the business changes direction, then that's where the most changes need to take effect, impacting all other services that depend on it.

As a methodology for architecture I quite like TOGAF. If you've read the book, you'll understand the role of an architect better and the place such a person takes in the organization. Within each organization, the politics are different though and thereby the expectations. It's always a challenge to find your right place and produce those results that make people happy. Sometimes you need to make some noise in order to wake up the masses.

As discussed in earlier posts, I strongly believe that taking more time for a design prevents a lot of troubles at a later stage. Troubles at a later stage are times more expensive than resolving that in the correct stage. Remember that if you're a project/program manager or director. You can't take shortcuts in IT. The taxman always rings the doorbell later on and will ring it more than once along different projects.

The latter statement basically means that a single shortcut taken now to get things done will eventually require interest payments later down the line that far exceed the initial amount and from my experience, this can be multiples of that. I've worked in a project that should have taken about 250,000 EUR or so to develop. It crossed the million by a good couple of thousands.

Further reading:

http://www.cis.gsu.edu/~mmoore/CIS3300/handouts/SciAmSept1994.html

Project Dune was released yesterday and I intend to re-evaluate software engineering from the ground up, first as a set of activities and practices rather than thinking of it as a process only. In another sense, it's as if the texts assume that people basically deliver good work, but as if each activity isn't always followed and each activity isn't bound to the next phase. I don't really believe that. I think that too many engineers are insufficiently aware of the impact of not executing certain activities or not following a certain best practice. The problem is mostly to do with pride, wanting to be creative and differentiating, or over-estimating their own set of skills (especially against fellow engineers in the same team). Better yet, it's difficult to come up with an experienced team of engineers that beyond knowing their profession, also know how to work together and get things done without introducing new unknown technologies (for the sake of curiosity and learning something new).

Maven

I've never had a good chance to look at maven, but just today I've for the first time generated the "project(code) management" layout through maven. I'm very impressed at its capabilities and dependency management and intend to use it for any project following up. It's just too good really and there's not much sense to keep hacking ant files in order to manage the project.

As an intro to maven... It's a great tool to start out a new project and manage the project configuration, build, dependencies, tests, releases and sites with. It does a lot of stuff out of the box in a standard way and you could more or less say that it's a 3rd-4th generation "make" tool. Where with any you still had to "program" tasks, with ant you can just configure pre-defined tasks that generally run in a consistent way (since every project needs it) and be done with it. Having a consistent layout also means that you don't need those ${src} and ${build} variables any longer.

It has the ability to produce manifests for jars, version/build numbers and can automatically deploy to locations you register. There is inheritance between configuration files, so that you can set up a chain of configuration files for your development "street" as you call it. And due to the registration and management of library repositories, your dependencies are automatically tracked and updated whenever the project thinks it needs them. No more looking for lib files and downloading them, it does it for you.

For Project Dune, there's an effort starting where we'll document software engineering in a very straightforward way. Rather than calling it a "methodology" like Scrum, XP, RUP and whatever, the idea is that we just focus on the activities that must happen anyway regardless of the approaches used to fulfill those activities.

Then the idea is that besides the activities, you might need the output of one activity as input to another or trace back information back to the source. You could imagine a certain "thread of information" running through a development life-cycle.

Every sw project has such a cycle, but not everybody commits to executing all the required activities to guarantee quality. For example, I imagine that RUP adopters execute a lot of activities, but not every activity necessarily adds value. And people working solo on a project may never do active reviews.

The idea is that the process doesn't become the holy grail of an organization (the administrative efforts), but the activities that are part of engineering. I don't have a full view yet how this is going to work out, but the site on project dune will be modeled after that vision. It's not very likely to get the same size as a process-driven description, but a minimal site where activities are clear and what they contribute to a project is already a very good start.

Work on roadmaps, sw engineering, technical debt

Project Dune is developing a new roadmap for the last half of 2008 and to kick off 2009. Being open-source, you can always read the latest developments on the project wiki.

Part of the roadmap now became strategy, which will probably be moved out to the start page close to the mission, thereby making the roadmap a practical guide of where something is going and why, without necessarily focusing too much on the actual activity.

Another part of the roadmap of Dune is the initiation of a set of processes, practices and tools / checklists / material to support the process of software engineering.

A particularly interesting term that was coined by Ward Cunningham is the term technical debt. As a quote from the wikipedia site:

The analogy to financial debt is that the poorly-written code requires "interest payments" of maintenance effort, which would be smaller or non-existent had the code been developed more carefully and with better long-term planning. Rearchitecting and rewriting the code to be more maintainable is analogous to paying off the debt principal.

Other sites:

http://www.agileadvice.com/archives/2006/12/technical_debt.html

http://forums.construx.com/blogs/stevemcc/archive/2007/11/01/technical-debt-2.aspx

The graphs here are very good examples of the consequences:
http://kanemar.com/2006/07/23/technical-debt-and-the-death-of-design-part-1/

As part of Project Dune, I'll need to think about how to come up with practical guidelines for software quality and explain these terms in more detail, probably with extensive use of links. The idea is to become the main site where people look for quality information. That should include links and understanding about software engineering, development, language choices, etc... Ideally documented in such a way that it goes from a high-level overview to a detailed overview.

The first objectives are to come up with a generic process / vision of software engineering itself. This is no easy feat! It's difficult enough to align people on the same vision within a single team as there are always people that have different opinions or people that have the thorough conviction that things are done better in a different method.

Thus, the idea is not to suggest "Agile Development" or any other more specific method for software engineering. It should go one level above all this and just state the following:

• Objectives of the activity
• The activity's function
  

These two sound exactly the same, but are somewhat different. The objective is basically what you're trying to achieve. The function is what you're trying to prevent/discard during that process or what you should pay specific attention to.

I'm wondering whether project managers in SW projects nowadays have sufficient knowledge of project analysis and the terms used for sw engineering to be able to steer back to a winning situation. If you look at the term "technical debt" for example, its actual scope is quite large. Besides technical debt though, there are other reasons why projects fail, which are more on the area of communication and social interaction (the requirements need to be correct).

A good initiative for the Dune project would be to try to come up with 2 or 3 main areas and coined terms that contribute to project failure. Then subdivide the areas and identify the causes.

The idea being that the identification of root and main causes for project failure (and supporting cases?) would clarify the need and use of certain activities in a project. I'm not using the term process, as I'm not too fond of that term how it's interpreted in general. Activity means something that a tester/developer carries out as part of his job or task and which is easily part of a process. The process is thus basically a sequence of activities done with a number of available tools and materials, nothing more. Understanding the process thus doesn't mean how to comply with some corporate or industrial standard. It means that you understand what you're doing and how that fits in the big picture.

Artificial Intelligence

I'm starting a studies on A.I. in September at the Free University in Amsterdam. It's a bit closer to Computer Science than other universities are offering.

It's interesting how I'm now picking up news articles on the topic and how things are changing around the very definition of what A.I. really is. Initially, the idea was that intelligent machines should eventually think like human beings and be just as versatile, flexible, inflexible and emotional and detached as humans can be.

It seems that the general definition of A.I. is changing in such a way that it's more a definition of very intelligently applying science to real-world problems that we can solve intelligently, but perhaps not optimally.

Thus, A.I. is not a simulation or replication of the entire scala of processes in the human brain, it's a specific simulation or replication of a specific problem that our brain can resolve.

As I have stated before in one of the blog posts on communication with machines, I think A.I. is about getting closer to the real cognitive processes that are occurring instead of replacing our tactile activities. Thus, A.I. is also a specific kind of Information Technology. It's about finding out how we resolve very difficult problems, the input to those problem resolution processes, applying science over increasing the efficiency and accuracy of the resolution of the problem and then deploying out so that it can be used by the user.

There are however some other specific branches. One branch comes closer to knowledge management (that is basically the problem of digging out the right information at the time when it's most needed). Another is about creating user interfaces and human interaction in different ways (more like emotional intelligence and empathy). Others are more about information processing on very large amounts of information.

EK & GWT

Holland is playing really well in the EK. It's a joy to watch. Got 6 points in the pocket and all of Europe is commenting how well-greased the team is playing this year. I'm expecting Holland to win this EK.

Well, for something totally different, a new version of GWT is out, 1.5. The beta version has good new capabilities and standard themes that prove very useful. I'll be plugging that into my project and keep improving the user interface. Then probably release 1.5.1 before I'm hosting a demo next month at the company I work for, in front of project managers, architects, testers and other people.

CUDA and GPU's

I've read a lot on the architecture of GPU's and how it's possible to take advantage of them. The design of a neural network running on a GPU is substantially different from those that run on single-core CPU's. I've worked on a small framework network to process neurons on a dual-core, but haven't been very impressed with the results so far.

The GPU however seems very promising. It actually requires a lot of threads and blocks to become efficient. My 8600GT has 128 little processors ready to work. The GPU basically organizes units of work into blocks, so that those processors can cooperate amongst little units of work. And those blocks are allocated threads, which could also be compared to the smallest unit of work within a block, a single iteration that is to be executed by that block.

The GPU of Nvidia is mostly data-parallel driven. You decide what you want to do and then run a very simple function by a single thread. The activation of a neuron by another neuron is an example of such a very simple function.

There are a lot of hardware-related optimizations that need to be taken into account. Ideally, the architecture of parallel systems may synchronize within blocks, but should never synchronize inbetween blocks themselves to prevent deadlock situations, plus that synchronization is a killer for performance.

The biggest problem for making graphics cards very useful for A.I. is the memory storage capacity *or* the bandwidth between the host memory / disk and the graphical card memory. It's basically 8 Gb/s on a standard PC with PCIe card, whilst internally on the card the bandwidth from it's memory to GPU is orders of magnitude higher, about 60 - 80 Gb/s. So staying on the card for calculations is definitely better for performance. The bandwidth to CPU memory is about 6.4 Gb/s by the way, so it's faster writing to the graphics card than reading/writing to its own memory.

The card contains 256MB of memory. If 4 bytes are used for various information needs like fatigue, threshold and excitation information, then it can store 67 million neurons on one card. It might be possible to use an extremely clever scheme to store connection information, because that information is where most memory is lost on. If you assume 1,000 connections per neuron, that is where 4,000 bytes of information are lost per neuron due to pointer size. Maybe a clever scheme where the neurons are repositioned after each cycle may help to reduce the need for such capacity.

Thus, assuming 4000 bytes for each neuron on average without such optimization or clever scheming, the network can only be 16,750 neurons in size at a maximum.

The design is the most interesting part of this challenge. Coding in CUDA isn't very special, although someone showed that from an initially 2x increase in processing power you can actually attain 20x increase in processing power if you really know how to optimize the CUDA code. So it's worth understanding the architecture of the GPU thoroughly, otherwise you're just hitting walls.

Kudos to CUDA

CUDA is an nvidia technology that allows programmers access to the processing power of GPU's, virtually turning your computer into a super-computer by loading certain complex mathematical processing on the GPU itself. Actually, a GPU is a collection of little processors of generally about 128, which are running functions in parallel. So, a GPU will probably benefit you most if you're processing a large dataset that fits into memory. For many simple business applications, it's not worth the trouble.

The GPU needs loads of different threads to become more efficient. The program thus slices up pieces of the algorithm to be processed and then executes its processors on it, that means the design of your 'program' will need to be adjusted accordingly. In comparison with a single CPU, if you have one processor the number of threads quickly saturate the processing power with thread-context switching overhead, thereby mitigating the results of the parallel processing. That's why it's better to have many little processors than one very large one.

The GPU can process algorithms about 250 times faster than a dual-core 2.4GHz CPU. So in other words, if you have a problem that is suitable for being loaded on the GPU, you can have access to a 250node CPU cluster by buying a commercial-grade graphics card of 250 euro's. That is very good value for money! And there are motherboards available where you can load 4 of those cards into your computer. You'll need a 1500W power supply though, but that is far less than 250 * (300/400)W. And there are guys at the university of Antwerp that have built such a supercomputer for the price of 4,000 Euros.

Here's a link to a tutorial on how to install this on Linux:

http://lifeofaprogrammergeek.blogspot.com/2008/05/cuda-development-in-ubuntu.html

One of the obvious applications why I'm looking at this is the ability to use the GPU for applications in neural network, more or less like what has been done here:

http://www.codeproject.com/KB/graphics/GPUNN.aspx

The memory bandwidth is in the order of 20-100 GBs/sec, as you can see here:

http://www.nvidia.com/page/geforce8.html

As I have stated in previous posts, one of the constraints of a working artificial network is the processing power that is available and the amount of memory storage due to the weights and so on that need to be coded. If things are hardware-accelerated, it might open up new avenues for processing. One of the larger problems remaining is probably the design of coding the network into simpler data elements most efficiently. Leaving things to one processor to figure out puts pressure on the frequency that can be developed. But if there are 128 processors instead, it's becoming very interesting indeed.

One of my posts suggested that I don't believe (yet) that there's a kind of output wire where the results of the network end up in, but rather that the state of the network itself, or sub-states of cell assemblies somehow form up consciousness. And the more neurons are available, the more of "consciousness" you'll have.

One should never forget that the idea of something having consciousness or awareness depends on our perception. And perception is always influenced by our own ideas and assumptions. So if we look at someone pretending to be intelligent, we also assume they have the same level of conscience that pertain to those tasks, or more even. But that's not necessarily an accurate assumption.

A probably mistake in a previous post is related to the frequency of the network. Probably, we can't really discuss 'frequency', as there's no clear cycle or sweep in the biological network if we assume that neurons fire whenever they want and the "sweep" that would collect all neurons that are currently activated might just come in somewhat randomly. Computers are generally programmed to define proper "work" cycles against a single cycle that collects the results.

The frequency of brain waves has been defined as between 5 to 40Hz. So the figure of 1000Hz may be way off. If we regard 40Hz, things may become a lot easier. And if we work with simpler processors that work on this frequency and on equal units of work, perhaps it brings things closer to the real human brain.

From the perspective of processing, the GPU would enable a calculation speed of 250 times that of a CPU. And if we lower the frequency from 1000 to 40 Hz, that is another multiplication factor of 25. That brings the new number of neurons that can be processed to 2,500,000,000. This is only a factor of 40 lower than the human brain!

Thus, if we put 40 graphics cards into a computer, we'd probably close to the processing power of the brain itself. That'd be about 8000W versus 12W. Not too bad, because using CPU's that'd be about 2,000,000W. The remaining problem is the storage of network information to be able to use the network. That was set at 2Gb and with some smarter designs and optimizations or reductions in the number of connections this could be brought down to 500M or so, so that a network of 500,000 neurons could run on a single graphics card, but it's not yet enough. A single byte is possibly sufficient though for a neuron and you wouldn't typically use a single byte for processing on a CPU due to byte-alignment optimizations. On the GPU that shouldn't matter too much though.

It's time to delve into the design of the processors and see what the processors of a GPU can deal with efficiently. If they work on bytes rather than words, it makes things a lot easier to fit in memory, thereby increasing the size of the network.

And then it doesn't matter too much. Perhaps SLI can help to distribute tasks across networks and may help to assign specific tasks to specific cards, like visual processing, reasoning and so on. Graphics cards generally use texture maps and so on and those can be extracted from cards and loaded onto others in an effort to share information.

Quantum consciousness, curling and glials...

The book of Prof. Penrose is a very interesting read. For the core A.I. people that believe in the "everything is perfectly computational and so is human thought" (not taking into account memory, processing or storage resources), it can be bit disheartening. Prof. Penrose asserts that there are things about the mind that can be simulated, although not perfectly, but that by simulating things it doesn't make a machine aware.

This of course depends on the definition of awareness, how awareness is raised in the human brain and whether awareness is real or whether we're just thinking we're aware and not in the same computed state as something else.

Another interesting thought here is with regards to the ability to steer thought and the ability to visualize things "in the mind's eye" as it's called. The interesting thing here is that besides regular perception that reacts to stimuli, we can also invoke stimuli on our own brain. Mostly for thought experiments or for dreaming. Although the last objective is nicer :).

I refer to the last post on "The plot thickens". I imagine that glial cells may have a more active role than previously perceived. Neuro-scientists have for example asserted that the glial cells are cleaners.

Ever heard of the sport "curling"? It's a sport where you have a heavy 20kg stone that is "thrown" on ice. The team consists of three players, where one player executes the throw (neuron) and two players clean the track in front of it. The cleaners can affect the trajectory of the stone, lengthening it or shortening it or moving it to a side and thereby affect the nature of the game.

One could imagine that the cleaners in curling have a similar role to the glial cells. If it's true, then it certainly makes things more interesting. And the search for the real "consciousness" would also just be starting. Would it be something in the brain? Would it be more like "waves"? Those glials than sort of become the manipulators of neuron cells, like an overwatch or teacher. If one maintains the Hebb's theory, the plot thickens indeed. Then learning isn't necessarily triggering neurons based on input and then making the cells fit, learning would be controlled by a network 10 times the size of the neurons, which seem to be driven by some other force.

At this point I'm not sure how to imagine consciousness then, nor do I intend very well the activity and power of the glial cells to influence the behaviour of neurons (or slight modifications in its firing pattern, or perhaps entire modification of the network itself?). Is this where consciousness is really located?

Some people resort to quantom consciousness.

The plot thickens...

The plot thickens as they say... And this time it's about glue!

Only since somewhat recently, scientists discovered that the glial cells around the neurons may be a bit more active than just removing waste and feeding the neurons. Read more about glial cells here.

Actually, there are 10-15 more glial cells than neurons. So if you thought 100 billion was a lot of cells, there's 10-15 times more of that in supporting cells.

The cells are mostly responsible for maintenance. So they regulate the chemicals, clean up waste, regulate blood supply. But they also produce a myelin sheath around the axon (helping it to fire) and can also act as scaffolding for the generation of new neurons.

So, you could say that glials are the implementation of the rules to create and maintain the network on a microscopic level. During its lifetime, it continuously checks the environment and can actually reshape the network on a local level. This is interesting, because then we know that there are complicated and active maintenance functions on the neural network taking place.

Neural Network Hierarchies

The book I read discussed the possibility of cell assemblies and cell assembly resonance through recursive loops in the neural network. It also stated the possibility of cooperating neural networks that are each allocated a specific function. See the following page:

http://faculty.washington.edu/chudler/functional.html

And then the following:

http://faculty.washington.edu/chudler/lang.html

I'm not sure if anyone has ever considered to join neural networks together in a sort of serial combination. The difference between these networks is that for example vision is only allocated the task of recognizing images / shapes / forms / colors and translate then into numbers. And the auditive system processes sounds. If you look at the images closely, you see that there are actually two kinds of networks for each perception method. An associative cortex and a primary cortex.

If you look at the production of speech, it's a number of different areas all working together. This gives us clues about how human speech is really put together.

Imagine those networks all working together. As a matter of fact, there are more neurons at work than just the brain. The eyes also have neurons and are already the first stage of processing optical information. Suppose we'd like to make a computer utter the word "circle" without simply recognizing the circle and play a wave file. We'd have to make it learn to do so:

• Convert the pixels (camera?) to a stream of digital information, which can be processed by the visual cortex.
  
• Analyze the shapes or image at the very center of the image (see motor reflexes and voluntary movement of the eye to accomplish the scanning of the environment for receiving more information).
  
• The visual cortex will then produce different signals as part of this exercise and the reverberating cell assemblies generate new input signals for the more complex processing of such information (memory? context?)
  
• This is then output to the speech area, where the "words" are selected to produce (mapping of signals to concepts).
• The information is then passed to the Broca area, where it is contextualized and put into grammar.
• The instructions of the Broca area (which could have a time-gating function and verifies the spoken word with the words that should be uttered), are sent to the primary motor cortex, which produces speech by frequent practice
• The speech organs move as in concert by the simple emission of information towards the speech organs.

The above sequence displays a very interesting point. Wernicke's area is involved in understanding heard words, Broca's area is involved in producing words.

So, this sequence shows that these areas work together and that together, the emergent phenomenon can produce very interesting behaviours.

I'm not sure if these networks can be built by just trying them out at random. There's also a huge problem with the verification of the validity of a network. We can only validate its validity whenever the output at the other side (hearing the word) makes sense due to the input (the visual image of a circle). Everything that happens inbetween can develop problems in this entire circle. Also, there is expectedly a very large learning curve required to produce the above scenario. Remember that children learn to speak only after about 1.5-2 years or so, and then only produce words like 'mama' / 'papa' (as if those words are embedded memory in DNA).

Important numbers and statements

The following are statements that are important to remember and re-assess for validity:

1. The brain consumes 12W of energy. Ideally, artificial simulations of the brain should respect this energy consumption level. But this seems far from possible because the individual elements used in artificial intelligence consume far more power and there are factor thousands involved in this calculation.
   
2. It should be parallel in nature, similar to neuron firings (thread initiations) that fire along dendrites and synapses. If not, the model should assess scheduling in a single thread of operation.
   
3. It should be stack-less and not have function unwinds.
4. The brain has about 100 billion neurons.
5. The fanout (connections) with other neurons is between 1,000 to 10,000, (others report this to be 20,000).
   
6. It's not so much the working neural network that is interesting, but the DNA or construction rules that generate the working neural network. That is, it's more interesting to come up with rules that determine how a network is to be built than build a network that works and not being able to reconstruct it elsewhere.
7. How to observe the state of the network in an intelligent way in order to deduce conclusions from the observation by the network?? (does it need to do so?)
   
8. It is possible that successfully working networks can only evolve / develop over a certain time period and that the initial results look like nothing interesting at all. This statement can be deepened out by observing the development of infants.
9. How does the state of a brain transcend into consciousness? (or is thinking the re-excitation of network assemblies by faking nerve input, imagination, so that images and audio seem to be there?)
   
10. Zero-point measurement: My computer (a dual intel E6850 with 2GB low-latency memory) can process 500,000,000 (500 million) neuron structures in 0.87 seconds. That is about 1.14 cycles per second on 500,000,000 neurons. That is still a factor of 100 * 1000 = 100,000 slower than the human brain, assuming it re-evaluates all neurons in one sweep.
    
11. For a very simple neuron structure on a 50 that does not yet contain connection information, but 3 bytes for threshold, fatigue and excitation information, 140 GB of memory is required to store this network in memory.
12. In 2 GB of memory, you can fit 715,000,000 neurons without connection information.
13. 50 billion neurons need 186404 GB of memory to store an average of 1,000 connections at a pointer size of 4 bytes per neuron.
14. On my CPU (E6850) and a single thread/process, a number of 400,000 can reasonably be processed in one sweep. That makes it about 1,500 sweeps per second across the entire neuron array.
15. In 2GB of memory, it's possible to fit 500,000 neurons with connection information.
    

I'm therefore choosing 500,000 neurons as the basis of the network, which might eventually translate to a frequency of about 1000Hz if the sweeps are designed more carefully (1000Hz is derived from extremely high firing rates in the human brain that are observed to be at 200 pulses per second. Add the absolute refractory period to that, which lasts 3-4 cycles, and 1000Hz emerges).

500,000 seems to be the limit due to memory and due to CPU cycles in order to attain the same frequency. That is a factor 100,000 lower than the human brain and it's more or less maxing out the machine.

The emergence of intelligence

John Holland wrote one of the most interesting books I've read so far, "Emergence". And it's not even the size of the Bible. :).

My previous musings on cognitive science and neural networks and artificial reasoning are greatly influenced by this book.

As I've stated in one of the posts on this blog, I've sketched out an argument that the "output-as-we-know-it" from artificial networks isn't so much useful from a reasoning perspective, but the state of the network tells a lot more about "meaning" than measuring output at output tendrils. I'm not sure whether for very complicated and very large neural networks you would even have a type of output.

The book "Emergence" provides a potential new view on this topic. It makes clear that feed-forward networks (as used in some A.I. implementations) cannot have indefinite memory. Indefinite memory is basically the ability of a network to start reverberating once it recognizes excitation at the input and further continuous excitation further on. The capabilities of a network without memory are greatly reduced and after reading the text, I dare say that pure feed-forward networks are very unlikely to be at the base of intelligence.

Indefinite memory is caused by feedback loops within the network. So you'd have a neuron that connects to some neuron of the previous input layer or a previous hidden layer, thereby increasing the likelihood it will fire in the next cycle.

There are however additional features required for a feedback network. It has a fatigue factor and a recently fired neuron has a very high threshold for firing again for a short time period. As neurons are continuously firing, these become fatigued and gradually decrease the likelihood it will fire in subsequent rounds. This helps to decrease the effect of continuous excitation (and may explain boredom). Plus that neurons that have just fired increase their threshold of firing significantly for the next couple of rounds (about 3-4), further decreasing the chances of reverberation across the network in a kind of epileptic state.

The end result for such a network are three important features: synchrony, anticipation and hierarchy. Synchrony means that certain neurons or cell assemblies in the network may start to reverberate together (through the loops), which is an important factor in anticipation, where cell assemblies reduce their thresholds of activation, so that they become more sensitive to certain potential patterns (it's as if the network anticipates something to be there, so it's the memory of where things might lead in some context), and hierarchy, where cell assemblies may excite other assemblies. The other assemblies may then represent a concept slightly higher in the hierarchy (for example a sentence as opposed to a word).

As has been discussed in the post on the implementation of humor, we can derive that humor is probably induced by the felt changes in the network (electricity and fast-changing reverberations to other cell assemblies) as the changes in context develop sudden changes in excitation across the network.

Thus, humor can be described as a recalibration of part of the network that is close enough to the original reverberation pattern, but not as distant as to become incomprehensible.

The final assumption I'm going to make then is that a certain state of the network (reverberating assemblies) correspond to a particular meaning. There is indeed a kind of anticipation in this network, and recently reverberated assemblies might reverberate very quickly again in the future (short-term memory).

Then perhaps memory is not so much concerned with remembering every trait and feature as it is observed, but more concerned with storing and creating paths of execution and cell assemblies throughout the network and make sure they reverberate when they're supposed to. Then memory isn't so much "putting a byte of memory into neuron A", but it's the reverberation of cell assemblies in different parts of the network. Categorization is then basically recognizing that certain cell assemblies are reverberating, thus detecting similarities. We've already shown that the effect of anticipation reduces the threshold of other assemblies to reverberate, although it doesn't necessarily excite them.

Question then is of course how the brain detects the assemblies that are reverberating? It requires a detector that has this knowledge around the entire brain in order for this theory to make any sense. As if it knows where activity is taking place around the network to induce a kind of meaning to it. The meaning doesn't need to be translated to words yet, it's just knowing that something looks like (or is exactly like) something seen before.

Actually, the interesting thing of memory is also that different paths can lead to the same excitation. So the smell of grass, the vision of grass, the word grass, the sound of grass and other representations may all be somehow connected.

In this thought-model, if we would form sentences by attaching nouns to reverberating assemblies, it may be possible to utter sounds from wave-forms attached to those concepts and perhaps use the path of context modification (how the reverberating assemblies shift to new parts) to choose the correct wording. Or actually, I can imagine that multiple assemblies are active at the same time, also modifying the context.

Multiple active assemblies seem like a more plausible suggestion. It would enable higher levels of classification in different ways, although it does not yet explain the ability of our mind to re-classify items based on new knowledge. Do we reshape our neural network so quickly? Although I must say that we do seem to make previous mistakes more often for a certain period of time until at some point we dislearn it and relearn it properly. Dislearning something has always been known as more difficult than learning something.

A very interesting thought here is the idea of the referee. If the network is allowed to reverberate in a specific state, how do we learn so effectively? We continuously seem to test our thought to reason and explanation of how it should be. Is there a separate neural network on the side which tests the state of the network against an expected state? That would however require two brains inside of one, and one to be perfect and correct to measure the output of the other, thereby invalidating that model. Perhaps the validity of the network can at some point be tested against its own tacit knowledge. Does it make sense that certain categories or cell assemblies reverberate in unison? If they have never done that before, then perhaps the incorrect conclusions are made, which should cause the network to discard the possibility, reduce the likelihood of reverberation of a certain cell assembly and keep looking for sensible co-reverberation.

To finalize the topic for now... Emergence requires a network of agents that interoperate together through a set of simple rules. The rules that I found most interesting for now are described in this blog post. But I can't help but wonder about the role of DNA. DNA is said to have its own memory and it's also known to represent a kind of blue-print. Recently, some researchers have stated that DNA isn't necessarily fixed and static, but that parts of DNA can become modified within a person's lifetime. That would be a very interesting discovery.

Anyway, if we take DNA as the building blocks for a person's shape, features and biological composition (besides the shape influences due to bad eating habits and so on), then we have certain body features that are controlled by DNA and probably certain human behaviour that is reflected in our children ( "he takes after him/her" ).

Just the recognition that behaviour can be transcended by children makes a strong case that the building up of the human brain is determined by rules that are prescribed by the DNA, a kind of "brain blue-printing", a recipe for how to build a brain through a set of rules.

So, we could create a neural network through very random rules and see what happens, but we could also think of the construction of that network to have followed certain rules that are determined through evolution. This would make a particular network more effective at each generation. It's a big question. Real connections are formed by neurons that just happen to be close by another and I cannot imagine the possibility that a neuron on one side of the brain manages to connect to a neuron at a significant distance.

Maybe the construction of this network is determined by a lower level of emergence, which is determined by smaller elements like DNA and whatever else is there at an organism level. Perhaps our consciousness starts with those minuscule elements?

Or just maybe the growth of the brain is entirely random. We could then consider the possibility that neurons exist somewhere and grow towards another. Then, through Hebb's rule, it might continuously attempt to reverberate and kill those axons between neurons that never lead to reverberation together (thus, have no useful interconnection with one another). Especially in the first four years, these connections (axons) grow like wildfire in a continuous manner. It takes four years for a network of 50 billion neurons to start producing some sensible results. We generally kick-start a network and almost expect it to produce something interesting after five minutes.

It would be very interesting research to find out if this kind of growth/evolution can be jump-started and done in much less of the time through application of a computer cluster (or whether the brain can run on clusters in the first place :).

On content and process

I read a very interesting post just recently regarding the difference of content versus process. Process is basically determining action based on the context and is very much done in the here and now. Content has to do with analysis of concepts and the relationships between them and could be taken as learning experiences. Process can also be learning, but the enhancement of action (reflex) on the perception of content identified in a way. Content itself is deep-rooted knowledge of how a concept might have gotten somewhere or how it might relate to other concepts (in various different possible ways).

Maybe if you don't appreciate the arts, you're a person that highly prefers process (objectives), getting things done or moving from A to B without caring much about the how and where. People that really dig art and content may not be as efficient in getting their things done, but they understand the relations between concepts better and "enjoy the journey" :).

This is an interesting differentation of course. If not achieving your objective causes frustration, than this might also explain why some people feel depressed, frustrated or stressed more than other people. Some are just there for the journey and the pleasure, others always want to be somewhere else just as they got somewhere.

The argument of the person writing the article was that long, continuous exposure to video games and films for example didn't train the content-analyzing capabilities sufficiently. Therefore, training people to only get things done without training them on the pleasure/enjoyment of analyzing the interrelationships and contents of things.

Historically, humans have mostly lived together and generally spent a lot of time interacting with one another, developing and improving interpersonal relationships. The virtual environments are however loaded with objectives "just to make it interesting", so the argument that social environments improve relationships isn't a natural argument. It might just provide an excuse for achieving your objectives.

The article also articulated that the development of the individual (the recognition of who you are yourself, your "self-idea" ) isn't as developed. Or stated in another way, you're not sufficiently self-aware or "individualawared" enough. This puts pressure on the need to make more effort to be recognized as a specific type of person, or projected self-image.

This lack of individuality could then be compensated by collecting status symbols, generally projected symbols of what is considered success by oneself. Those symbols of success are basically material trophies like cars, houses and other things, material things that are thought to add up to one's identity. Sadly to say though, one can never gather enough items for individualization, there's always place for more "self-articulation", which explains the unexhaustive search for new items to conquer.

Cocomo II implementation on Project Dune

As discussed in previous posts, I've been designing an approach for Cocomo II software project estimations. The implementation has already been added to Project Dune on the main branch and is destined for finalization into a new major version of the project, Project Dune 2.0.

The first step in estimation is to determine the project size:

http://gtoonstra.googlepages.com/sizing.png

The next step in this approach is to determine the effort multipliers. Those multipliers increase the time required to develop something linearly.

http://gtoonstra.googlepages.com/effortadjustment.png

The scaling factors in the next tab increase effort exponentially, so if those end up really high, the effort required is soaring as well:

http://gtoonstra.googlepages.com/scalefactors.png

After the factors are supplied, the outcome of Cocomo II is a set of numbers. The most useful numbers at this time are listed here:

http://gtoonstra.googlepages.com/result.png

Notice in the different screenshots how the factor being manipulated is explained in the tooltip. The text in the middle is a reflection of the correct assessment of that factor.

The results show the person-months that are required to develop the project, the nominal person-months (if all scale factors and multipliers would be nominal), and the Time-To-Develop together with the number of staff required to implement the project.