Software Architecture "methodology"

As a software architect, buzzwords and the buzz itself comes at you in various guises. Sometimes it's the business itself that heard about company X employing methodology Y and they want to do the same. Most of the times it's a hotshot from company Z that wants to change the world and do things radically different, promising totally different flexibility and capabilities.

I'm a bit tired of all these words. The only thing that counts are activities, as they produce the output that can really be used, and revolving around that output are only a number of factors. No single methodology will ever succeed in setting up a dynamic company. It's the people themselves, how they interact and communicate, the level of politics, the culture and how quickly the development team picks up new technologies that are really worthwhile.

Architecture methodologies are basically all about the same thing... guidelines about how to execute the development activities and a registration of the foreseen limitations and recommendations for that team.

The latest thing is of course SOA. It's often thought of as a new methodology, but it's actually just a different perspective on your organization. It cannot be called an architecture though. It does force systems and data to be organized a certain way, but it does not dictate how. So, SOA is a direction that a company decides to take, which might make sense, but it isn't anything else than that.

I see EAI architectures at the core of architecture, influenced by SOA, whereas SOA is the organization of the infrastructure into a set of services. It's as if SOA is basically a aggregation of resources to make it an artificial and digital interface that other organizations can use. So, SOA is more like a vertical whereas EAI definitions are mostly describing the trees of the "what-otherwise-would-be" forest of resources. EAI defines the guidelines in getting the resources set up, SOA defines what to expose. EAI is driven by the output of SOA and SOA is limited by the level of organization in EAI.

Once you think about it, the only thing that changes are the levels of connectivity and the level of digitalization. Organizations in the 60's were already capable of exchanging information, it just mostly traveled on paper. Currently, the velocity and size of information has grown so much that paper processing is just not an option any more. That is why the complexity of security and interpretation is now being born by digital systems, whereas this previously was interpreted and verified by human beings.

So, although SOA makes it really interesting for companies, it's far from a silver bullet, although pulling off a successful infrastructure of that kind will very likely make you more flexible. The problem is however that in every architecture, a number of assumptions are always baked in. Sometimes, when businesses significantly change, the architecture can't change along with it, thereby requiring large investments to bring about that change. Every business and organization has a "heart" to it where data is kept or basic business processing is done. If the business changes direction, then that's where the most changes need to take effect, impacting all other services that depend on it.

As a methodology for architecture I quite like TOGAF. If you've read the book, you'll understand the role of an architect better and the place such a person takes in the organization. Within each organization, the politics are different though and thereby the expectations. It's always a challenge to find your right place and produce those results that make people happy. Sometimes you need to make some noise in order to wake up the masses.

As discussed in earlier posts, I strongly believe that taking more time for a design prevents a lot of troubles at a later stage. Troubles at a later stage are times more expensive than resolving that in the correct stage. Remember that if you're a project/program manager or director. You can't take shortcuts in IT. The taxman always rings the doorbell later on and will ring it more than once along different projects.

The latter statement basically means that a single shortcut taken now to get things done will eventually require interest payments later down the line that far exceed the initial amount and from my experience, this can be multiples of that. I've worked in a project that should have taken about 250,000 EUR or so to develop. It crossed the million by a good couple of thousands.

Further reading:

http://www.cis.gsu.edu/~mmoore/CIS3300/handouts/SciAmSept1994.html

Project Dune was released yesterday and I intend to re-evaluate software engineering from the ground up, first as a set of activities and practices rather than thinking of it as a process only. In another sense, it's as if the texts assume that people basically deliver good work, but as if each activity isn't always followed and each activity isn't bound to the next phase. I don't really believe that. I think that too many engineers are insufficiently aware of the impact of not executing certain activities or not following a certain best practice. The problem is mostly to do with pride, wanting to be creative and differentiating, or over-estimating their own set of skills (especially against fellow engineers in the same team). Better yet, it's difficult to come up with an experienced team of engineers that beyond knowing their profession, also know how to work together and get things done without introducing new unknown technologies (for the sake of curiosity and learning something new).

Maven

I've never had a good chance to look at maven, but just today I've for the first time generated the "project(code) management" layout through maven. I'm very impressed at its capabilities and dependency management and intend to use it for any project following up. It's just too good really and there's not much sense to keep hacking ant files in order to manage the project.

As an intro to maven... It's a great tool to start out a new project and manage the project configuration, build, dependencies, tests, releases and sites with. It does a lot of stuff out of the box in a standard way and you could more or less say that it's a 3rd-4th generation "make" tool. Where with any you still had to "program" tasks, with ant you can just configure pre-defined tasks that generally run in a consistent way (since every project needs it) and be done with it. Having a consistent layout also means that you don't need those ${src} and ${build} variables any longer.

It has the ability to produce manifests for jars, version/build numbers and can automatically deploy to locations you register. There is inheritance between configuration files, so that you can set up a chain of configuration files for your development "street" as you call it. And due to the registration and management of library repositories, your dependencies are automatically tracked and updated whenever the project thinks it needs them. No more looking for lib files and downloading them, it does it for you.

For Project Dune, there's an effort starting where we'll document software engineering in a very straightforward way. Rather than calling it a "methodology" like Scrum, XP, RUP and whatever, the idea is that we just focus on the activities that must happen anyway regardless of the approaches used to fulfill those activities.

Then the idea is that besides the activities, you might need the output of one activity as input to another or trace back information back to the source. You could imagine a certain "thread of information" running through a development life-cycle.

Every sw project has such a cycle, but not everybody commits to executing all the required activities to guarantee quality. For example, I imagine that RUP adopters execute a lot of activities, but not every activity necessarily adds value. And people working solo on a project may never do active reviews.

The idea is that the process doesn't become the holy grail of an organization (the administrative efforts), but the activities that are part of engineering. I don't have a full view yet how this is going to work out, but the site on project dune will be modeled after that vision. It's not very likely to get the same size as a process-driven description, but a minimal site where activities are clear and what they contribute to a project is already a very good start.

Work on roadmaps, sw engineering, technical debt

Project Dune is developing a new roadmap for the last half of 2008 and to kick off 2009. Being open-source, you can always read the latest developments on the project wiki.

Part of the roadmap now became strategy, which will probably be moved out to the start page close to the mission, thereby making the roadmap a practical guide of where something is going and why, without necessarily focusing too much on the actual activity.

Another part of the roadmap of Dune is the initiation of a set of processes, practices and tools / checklists / material to support the process of software engineering.

A particularly interesting term that was coined by Ward Cunningham is the term technical debt. As a quote from the wikipedia site:

The analogy to financial debt is that the poorly-written code requires "interest payments" of maintenance effort, which would be smaller or non-existent had the code been developed more carefully and with better long-term planning. Rearchitecting and rewriting the code to be more maintainable is analogous to paying off the debt principal.

Other sites:

http://www.agileadvice.com/archives/2006/12/technical_debt.html

http://forums.construx.com/blogs/stevemcc/archive/2007/11/01/technical-debt-2.aspx

The graphs here are very good examples of the consequences:
http://kanemar.com/2006/07/23/technical-debt-and-the-death-of-design-part-1/

As part of Project Dune, I'll need to think about how to come up with practical guidelines for software quality and explain these terms in more detail, probably with extensive use of links. The idea is to become the main site where people look for quality information. That should include links and understanding about software engineering, development, language choices, etc... Ideally documented in such a way that it goes from a high-level overview to a detailed overview.

The first objectives are to come up with a generic process / vision of software engineering itself. This is no easy feat! It's difficult enough to align people on the same vision within a single team as there are always people that have different opinions or people that have the thorough conviction that things are done better in a different method.

Thus, the idea is not to suggest "Agile Development" or any other more specific method for software engineering. It should go one level above all this and just state the following:

• Objectives of the activity
• The activity's function
  

These two sound exactly the same, but are somewhat different. The objective is basically what you're trying to achieve. The function is what you're trying to prevent/discard during that process or what you should pay specific attention to.

I'm wondering whether project managers in SW projects nowadays have sufficient knowledge of project analysis and the terms used for sw engineering to be able to steer back to a winning situation. If you look at the term "technical debt" for example, its actual scope is quite large. Besides technical debt though, there are other reasons why projects fail, which are more on the area of communication and social interaction (the requirements need to be correct).

A good initiative for the Dune project would be to try to come up with 2 or 3 main areas and coined terms that contribute to project failure. Then subdivide the areas and identify the causes.

The idea being that the identification of root and main causes for project failure (and supporting cases?) would clarify the need and use of certain activities in a project. I'm not using the term process, as I'm not too fond of that term how it's interpreted in general. Activity means something that a tester/developer carries out as part of his job or task and which is easily part of a process. The process is thus basically a sequence of activities done with a number of available tools and materials, nothing more. Understanding the process thus doesn't mean how to comply with some corporate or industrial standard. It means that you understand what you're doing and how that fits in the big picture.

Artificial Intelligence

I'm starting a studies on A.I. in September at the Free University in Amsterdam. It's a bit closer to Computer Science than other universities are offering.

It's interesting how I'm now picking up news articles on the topic and how things are changing around the very definition of what A.I. really is. Initially, the idea was that intelligent machines should eventually think like human beings and be just as versatile, flexible, inflexible and emotional and detached as humans can be.

It seems that the general definition of A.I. is changing in such a way that it's more a definition of very intelligently applying science to real-world problems that we can solve intelligently, but perhaps not optimally.

Thus, A.I. is not a simulation or replication of the entire scala of processes in the human brain, it's a specific simulation or replication of a specific problem that our brain can resolve.

As I have stated before in one of the blog posts on communication with machines, I think A.I. is about getting closer to the real cognitive processes that are occurring instead of replacing our tactile activities. Thus, A.I. is also a specific kind of Information Technology. It's about finding out how we resolve very difficult problems, the input to those problem resolution processes, applying science over increasing the efficiency and accuracy of the resolution of the problem and then deploying out so that it can be used by the user.

There are however some other specific branches. One branch comes closer to knowledge management (that is basically the problem of digging out the right information at the time when it's most needed). Another is about creating user interfaces and human interaction in different ways (more like emotional intelligence and empathy). Others are more about information processing on very large amounts of information.

EK & GWT

Holland is playing really well in the EK. It's a joy to watch. Got 6 points in the pocket and all of Europe is commenting how well-greased the team is playing this year. I'm expecting Holland to win this EK.

Well, for something totally different, a new version of GWT is out, 1.5. The beta version has good new capabilities and standard themes that prove very useful. I'll be plugging that into my project and keep improving the user interface. Then probably release 1.5.1 before I'm hosting a demo next month at the company I work for, in front of project managers, architects, testers and other people.

CUDA and GPU's

I've read a lot on the architecture of GPU's and how it's possible to take advantage of them. The design of a neural network running on a GPU is substantially different from those that run on single-core CPU's. I've worked on a small framework network to process neurons on a dual-core, but haven't been very impressed with the results so far.

The GPU however seems very promising. It actually requires a lot of threads and blocks to become efficient. My 8600GT has 128 little processors ready to work. The GPU basically organizes units of work into blocks, so that those processors can cooperate amongst little units of work. And those blocks are allocated threads, which could also be compared to the smallest unit of work within a block, a single iteration that is to be executed by that block.

The GPU of Nvidia is mostly data-parallel driven. You decide what you want to do and then run a very simple function by a single thread. The activation of a neuron by another neuron is an example of such a very simple function.

There are a lot of hardware-related optimizations that need to be taken into account. Ideally, the architecture of parallel systems may synchronize within blocks, but should never synchronize inbetween blocks themselves to prevent deadlock situations, plus that synchronization is a killer for performance.

The biggest problem for making graphics cards very useful for A.I. is the memory storage capacity *or* the bandwidth between the host memory / disk and the graphical card memory. It's basically 8 Gb/s on a standard PC with PCIe card, whilst internally on the card the bandwidth from it's memory to GPU is orders of magnitude higher, about 60 - 80 Gb/s. So staying on the card for calculations is definitely better for performance. The bandwidth to CPU memory is about 6.4 Gb/s by the way, so it's faster writing to the graphics card than reading/writing to its own memory.

The card contains 256MB of memory. If 4 bytes are used for various information needs like fatigue, threshold and excitation information, then it can store 67 million neurons on one card. It might be possible to use an extremely clever scheme to store connection information, because that information is where most memory is lost on. If you assume 1,000 connections per neuron, that is where 4,000 bytes of information are lost per neuron due to pointer size. Maybe a clever scheme where the neurons are repositioned after each cycle may help to reduce the need for such capacity.

Thus, assuming 4000 bytes for each neuron on average without such optimization or clever scheming, the network can only be 16,750 neurons in size at a maximum.

The design is the most interesting part of this challenge. Coding in CUDA isn't very special, although someone showed that from an initially 2x increase in processing power you can actually attain 20x increase in processing power if you really know how to optimize the CUDA code. So it's worth understanding the architecture of the GPU thoroughly, otherwise you're just hitting walls.

Kudos to CUDA

CUDA is an nvidia technology that allows programmers access to the processing power of GPU's, virtually turning your computer into a super-computer by loading certain complex mathematical processing on the GPU itself. Actually, a GPU is a collection of little processors of generally about 128, which are running functions in parallel. So, a GPU will probably benefit you most if you're processing a large dataset that fits into memory. For many simple business applications, it's not worth the trouble.

The GPU needs loads of different threads to become more efficient. The program thus slices up pieces of the algorithm to be processed and then executes its processors on it, that means the design of your 'program' will need to be adjusted accordingly. In comparison with a single CPU, if you have one processor the number of threads quickly saturate the processing power with thread-context switching overhead, thereby mitigating the results of the parallel processing. That's why it's better to have many little processors than one very large one.

The GPU can process algorithms about 250 times faster than a dual-core 2.4GHz CPU. So in other words, if you have a problem that is suitable for being loaded on the GPU, you can have access to a 250node CPU cluster by buying a commercial-grade graphics card of 250 euro's. That is very good value for money! And there are motherboards available where you can load 4 of those cards into your computer. You'll need a 1500W power supply though, but that is far less than 250 * (300/400)W. And there are guys at the university of Antwerp that have built such a supercomputer for the price of 4,000 Euros.

Here's a link to a tutorial on how to install this on Linux:

http://lifeofaprogrammergeek.blogspot.com/2008/05/cuda-development-in-ubuntu.html

One of the obvious applications why I'm looking at this is the ability to use the GPU for applications in neural network, more or less like what has been done here:

http://www.codeproject.com/KB/graphics/GPUNN.aspx

The memory bandwidth is in the order of 20-100 GBs/sec, as you can see here:

http://www.nvidia.com/page/geforce8.html

As I have stated in previous posts, one of the constraints of a working artificial network is the processing power that is available and the amount of memory storage due to the weights and so on that need to be coded. If things are hardware-accelerated, it might open up new avenues for processing. One of the larger problems remaining is probably the design of coding the network into simpler data elements most efficiently. Leaving things to one processor to figure out puts pressure on the frequency that can be developed. But if there are 128 processors instead, it's becoming very interesting indeed.

One of my posts suggested that I don't believe (yet) that there's a kind of output wire where the results of the network end up in, but rather that the state of the network itself, or sub-states of cell assemblies somehow form up consciousness. And the more neurons are available, the more of "consciousness" you'll have.

One should never forget that the idea of something having consciousness or awareness depends on our perception. And perception is always influenced by our own ideas and assumptions. So if we look at someone pretending to be intelligent, we also assume they have the same level of conscience that pertain to those tasks, or more even. But that's not necessarily an accurate assumption.

A probably mistake in a previous post is related to the frequency of the network. Probably, we can't really discuss 'frequency', as there's no clear cycle or sweep in the biological network if we assume that neurons fire whenever they want and the "sweep" that would collect all neurons that are currently activated might just come in somewhat randomly. Computers are generally programmed to define proper "work" cycles against a single cycle that collects the results.

The frequency of brain waves has been defined as between 5 to 40Hz. So the figure of 1000Hz may be way off. If we regard 40Hz, things may become a lot easier. And if we work with simpler processors that work on this frequency and on equal units of work, perhaps it brings things closer to the real human brain.

From the perspective of processing, the GPU would enable a calculation speed of 250 times that of a CPU. And if we lower the frequency from 1000 to 40 Hz, that is another multiplication factor of 25. That brings the new number of neurons that can be processed to 2,500,000,000. This is only a factor of 40 lower than the human brain!

Thus, if we put 40 graphics cards into a computer, we'd probably close to the processing power of the brain itself. That'd be about 8000W versus 12W. Not too bad, because using CPU's that'd be about 2,000,000W. The remaining problem is the storage of network information to be able to use the network. That was set at 2Gb and with some smarter designs and optimizations or reductions in the number of connections this could be brought down to 500M or so, so that a network of 500,000 neurons could run on a single graphics card, but it's not yet enough. A single byte is possibly sufficient though for a neuron and you wouldn't typically use a single byte for processing on a CPU due to byte-alignment optimizations. On the GPU that shouldn't matter too much though.

It's time to delve into the design of the processors and see what the processors of a GPU can deal with efficiently. If they work on bytes rather than words, it makes things a lot easier to fit in memory, thereby increasing the size of the network.

And then it doesn't matter too much. Perhaps SLI can help to distribute tasks across networks and may help to assign specific tasks to specific cards, like visual processing, reasoning and so on. Graphics cards generally use texture maps and so on and those can be extracted from cards and loaded onto others in an effort to share information.

Quantum consciousness, curling and glials...

The book of Prof. Penrose is a very interesting read. For the core A.I. people that believe in the "everything is perfectly computational and so is human thought" (not taking into account memory, processing or storage resources), it can be bit disheartening. Prof. Penrose asserts that there are things about the mind that can be simulated, although not perfectly, but that by simulating things it doesn't make a machine aware.

This of course depends on the definition of awareness, how awareness is raised in the human brain and whether awareness is real or whether we're just thinking we're aware and not in the same computed state as something else.

Another interesting thought here is with regards to the ability to steer thought and the ability to visualize things "in the mind's eye" as it's called. The interesting thing here is that besides regular perception that reacts to stimuli, we can also invoke stimuli on our own brain. Mostly for thought experiments or for dreaming. Although the last objective is nicer :).

I refer to the last post on "The plot thickens". I imagine that glial cells may have a more active role than previously perceived. Neuro-scientists have for example asserted that the glial cells are cleaners.

Ever heard of the sport "curling"? It's a sport where you have a heavy 20kg stone that is "thrown" on ice. The team consists of three players, where one player executes the throw (neuron) and two players clean the track in front of it. The cleaners can affect the trajectory of the stone, lengthening it or shortening it or moving it to a side and thereby affect the nature of the game.

One could imagine that the cleaners in curling have a similar role to the glial cells. If it's true, then it certainly makes things more interesting. And the search for the real "consciousness" would also just be starting. Would it be something in the brain? Would it be more like "waves"? Those glials than sort of become the manipulators of neuron cells, like an overwatch or teacher. If one maintains the Hebb's theory, the plot thickens indeed. Then learning isn't necessarily triggering neurons based on input and then making the cells fit, learning would be controlled by a network 10 times the size of the neurons, which seem to be driven by some other force.

At this point I'm not sure how to imagine consciousness then, nor do I intend very well the activity and power of the glial cells to influence the behaviour of neurons (or slight modifications in its firing pattern, or perhaps entire modification of the network itself?). Is this where consciousness is really located?

Some people resort to quantom consciousness.

The plot thickens...

The plot thickens as they say... And this time it's about glue!

Only since somewhat recently, scientists discovered that the glial cells around the neurons may be a bit more active than just removing waste and feeding the neurons. Read more about glial cells here.

Actually, there are 10-15 more glial cells than neurons. So if you thought 100 billion was a lot of cells, there's 10-15 times more of that in supporting cells.

The cells are mostly responsible for maintenance. So they regulate the chemicals, clean up waste, regulate blood supply. But they also produce a myelin sheath around the axon (helping it to fire) and can also act as scaffolding for the generation of new neurons.

So, you could say that glials are the implementation of the rules to create and maintain the network on a microscopic level. During its lifetime, it continuously checks the environment and can actually reshape the network on a local level. This is interesting, because then we know that there are complicated and active maintenance functions on the neural network taking place.

Neural Network Hierarchies

The book I read discussed the possibility of cell assemblies and cell assembly resonance through recursive loops in the neural network. It also stated the possibility of cooperating neural networks that are each allocated a specific function. See the following page:

http://faculty.washington.edu/chudler/functional.html

And then the following:

http://faculty.washington.edu/chudler/lang.html

I'm not sure if anyone has ever considered to join neural networks together in a sort of serial combination. The difference between these networks is that for example vision is only allocated the task of recognizing images / shapes / forms / colors and translate then into numbers. And the auditive system processes sounds. If you look at the images closely, you see that there are actually two kinds of networks for each perception method. An associative cortex and a primary cortex.

If you look at the production of speech, it's a number of different areas all working together. This gives us clues about how human speech is really put together.

Imagine those networks all working together. As a matter of fact, there are more neurons at work than just the brain. The eyes also have neurons and are already the first stage of processing optical information. Suppose we'd like to make a computer utter the word "circle" without simply recognizing the circle and play a wave file. We'd have to make it learn to do so:

• Convert the pixels (camera?) to a stream of digital information, which can be processed by the visual cortex.
  
• Analyze the shapes or image at the very center of the image (see motor reflexes and voluntary movement of the eye to accomplish the scanning of the environment for receiving more information).
  
• The visual cortex will then produce different signals as part of this exercise and the reverberating cell assemblies generate new input signals for the more complex processing of such information (memory? context?)
  
• This is then output to the speech area, where the "words" are selected to produce (mapping of signals to concepts).
• The information is then passed to the Broca area, where it is contextualized and put into grammar.
• The instructions of the Broca area (which could have a time-gating function and verifies the spoken word with the words that should be uttered), are sent to the primary motor cortex, which produces speech by frequent practice
• The speech organs move as in concert by the simple emission of information towards the speech organs.

The above sequence displays a very interesting point. Wernicke's area is involved in understanding heard words, Broca's area is involved in producing words.

So, this sequence shows that these areas work together and that together, the emergent phenomenon can produce very interesting behaviours.

I'm not sure if these networks can be built by just trying them out at random. There's also a huge problem with the verification of the validity of a network. We can only validate its validity whenever the output at the other side (hearing the word) makes sense due to the input (the visual image of a circle). Everything that happens inbetween can develop problems in this entire circle. Also, there is expectedly a very large learning curve required to produce the above scenario. Remember that children learn to speak only after about 1.5-2 years or so, and then only produce words like 'mama' / 'papa' (as if those words are embedded memory in DNA).

Important numbers and statements

The following are statements that are important to remember and re-assess for validity:

1. The brain consumes 12W of energy. Ideally, artificial simulations of the brain should respect this energy consumption level. But this seems far from possible because the individual elements used in artificial intelligence consume far more power and there are factor thousands involved in this calculation.
   
2. It should be parallel in nature, similar to neuron firings (thread initiations) that fire along dendrites and synapses. If not, the model should assess scheduling in a single thread of operation.
   
3. It should be stack-less and not have function unwinds.
4. The brain has about 100 billion neurons.
5. The fanout (connections) with other neurons is between 1,000 to 10,000, (others report this to be 20,000).
   
6. It's not so much the working neural network that is interesting, but the DNA or construction rules that generate the working neural network. That is, it's more interesting to come up with rules that determine how a network is to be built than build a network that works and not being able to reconstruct it elsewhere.
7. How to observe the state of the network in an intelligent way in order to deduce conclusions from the observation by the network?? (does it need to do so?)
   
8. It is possible that successfully working networks can only evolve / develop over a certain time period and that the initial results look like nothing interesting at all. This statement can be deepened out by observing the development of infants.
9. How does the state of a brain transcend into consciousness? (or is thinking the re-excitation of network assemblies by faking nerve input, imagination, so that images and audio seem to be there?)
   
10. Zero-point measurement: My computer (a dual intel E6850 with 2GB low-latency memory) can process 500,000,000 (500 million) neuron structures in 0.87 seconds. That is about 1.14 cycles per second on 500,000,000 neurons. That is still a factor of 100 * 1000 = 100,000 slower than the human brain, assuming it re-evaluates all neurons in one sweep.
    
11. For a very simple neuron structure on a 50 that does not yet contain connection information, but 3 bytes for threshold, fatigue and excitation information, 140 GB of memory is required to store this network in memory.
12. In 2 GB of memory, you can fit 715,000,000 neurons without connection information.
13. 50 billion neurons need 186404 GB of memory to store an average of 1,000 connections at a pointer size of 4 bytes per neuron.
14. On my CPU (E6850) and a single thread/process, a number of 400,000 can reasonably be processed in one sweep. That makes it about 1,500 sweeps per second across the entire neuron array.
15. In 2GB of memory, it's possible to fit 500,000 neurons with connection information.
    

I'm therefore choosing 500,000 neurons as the basis of the network, which might eventually translate to a frequency of about 1000Hz if the sweeps are designed more carefully (1000Hz is derived from extremely high firing rates in the human brain that are observed to be at 200 pulses per second. Add the absolute refractory period to that, which lasts 3-4 cycles, and 1000Hz emerges).

500,000 seems to be the limit due to memory and due to CPU cycles in order to attain the same frequency. That is a factor 100,000 lower than the human brain and it's more or less maxing out the machine.

The emergence of intelligence

John Holland wrote one of the most interesting books I've read so far, "Emergence". And it's not even the size of the Bible. :).

My previous musings on cognitive science and neural networks and artificial reasoning are greatly influenced by this book.

As I've stated in one of the posts on this blog, I've sketched out an argument that the "output-as-we-know-it" from artificial networks isn't so much useful from a reasoning perspective, but the state of the network tells a lot more about "meaning" than measuring output at output tendrils. I'm not sure whether for very complicated and very large neural networks you would even have a type of output.

The book "Emergence" provides a potential new view on this topic. It makes clear that feed-forward networks (as used in some A.I. implementations) cannot have indefinite memory. Indefinite memory is basically the ability of a network to start reverberating once it recognizes excitation at the input and further continuous excitation further on. The capabilities of a network without memory are greatly reduced and after reading the text, I dare say that pure feed-forward networks are very unlikely to be at the base of intelligence.

Indefinite memory is caused by feedback loops within the network. So you'd have a neuron that connects to some neuron of the previous input layer or a previous hidden layer, thereby increasing the likelihood it will fire in the next cycle.

There are however additional features required for a feedback network. It has a fatigue factor and a recently fired neuron has a very high threshold for firing again for a short time period. As neurons are continuously firing, these become fatigued and gradually decrease the likelihood it will fire in subsequent rounds. This helps to decrease the effect of continuous excitation (and may explain boredom). Plus that neurons that have just fired increase their threshold of firing significantly for the next couple of rounds (about 3-4), further decreasing the chances of reverberation across the network in a kind of epileptic state.

The end result for such a network are three important features: synchrony, anticipation and hierarchy. Synchrony means that certain neurons or cell assemblies in the network may start to reverberate together (through the loops), which is an important factor in anticipation, where cell assemblies reduce their thresholds of activation, so that they become more sensitive to certain potential patterns (it's as if the network anticipates something to be there, so it's the memory of where things might lead in some context), and hierarchy, where cell assemblies may excite other assemblies. The other assemblies may then represent a concept slightly higher in the hierarchy (for example a sentence as opposed to a word).

As has been discussed in the post on the implementation of humor, we can derive that humor is probably induced by the felt changes in the network (electricity and fast-changing reverberations to other cell assemblies) as the changes in context develop sudden changes in excitation across the network.

Thus, humor can be described as a recalibration of part of the network that is close enough to the original reverberation pattern, but not as distant as to become incomprehensible.

The final assumption I'm going to make then is that a certain state of the network (reverberating assemblies) correspond to a particular meaning. There is indeed a kind of anticipation in this network, and recently reverberated assemblies might reverberate very quickly again in the future (short-term memory).

Then perhaps memory is not so much concerned with remembering every trait and feature as it is observed, but more concerned with storing and creating paths of execution and cell assemblies throughout the network and make sure they reverberate when they're supposed to. Then memory isn't so much "putting a byte of memory into neuron A", but it's the reverberation of cell assemblies in different parts of the network. Categorization is then basically recognizing that certain cell assemblies are reverberating, thus detecting similarities. We've already shown that the effect of anticipation reduces the threshold of other assemblies to reverberate, although it doesn't necessarily excite them.

Question then is of course how the brain detects the assemblies that are reverberating? It requires a detector that has this knowledge around the entire brain in order for this theory to make any sense. As if it knows where activity is taking place around the network to induce a kind of meaning to it. The meaning doesn't need to be translated to words yet, it's just knowing that something looks like (or is exactly like) something seen before.

Actually, the interesting thing of memory is also that different paths can lead to the same excitation. So the smell of grass, the vision of grass, the word grass, the sound of grass and other representations may all be somehow connected.

In this thought-model, if we would form sentences by attaching nouns to reverberating assemblies, it may be possible to utter sounds from wave-forms attached to those concepts and perhaps use the path of context modification (how the reverberating assemblies shift to new parts) to choose the correct wording. Or actually, I can imagine that multiple assemblies are active at the same time, also modifying the context.

Multiple active assemblies seem like a more plausible suggestion. It would enable higher levels of classification in different ways, although it does not yet explain the ability of our mind to re-classify items based on new knowledge. Do we reshape our neural network so quickly? Although I must say that we do seem to make previous mistakes more often for a certain period of time until at some point we dislearn it and relearn it properly. Dislearning something has always been known as more difficult than learning something.

A very interesting thought here is the idea of the referee. If the network is allowed to reverberate in a specific state, how do we learn so effectively? We continuously seem to test our thought to reason and explanation of how it should be. Is there a separate neural network on the side which tests the state of the network against an expected state? That would however require two brains inside of one, and one to be perfect and correct to measure the output of the other, thereby invalidating that model. Perhaps the validity of the network can at some point be tested against its own tacit knowledge. Does it make sense that certain categories or cell assemblies reverberate in unison? If they have never done that before, then perhaps the incorrect conclusions are made, which should cause the network to discard the possibility, reduce the likelihood of reverberation of a certain cell assembly and keep looking for sensible co-reverberation.

To finalize the topic for now... Emergence requires a network of agents that interoperate together through a set of simple rules. The rules that I found most interesting for now are described in this blog post. But I can't help but wonder about the role of DNA. DNA is said to have its own memory and it's also known to represent a kind of blue-print. Recently, some researchers have stated that DNA isn't necessarily fixed and static, but that parts of DNA can become modified within a person's lifetime. That would be a very interesting discovery.

Anyway, if we take DNA as the building blocks for a person's shape, features and biological composition (besides the shape influences due to bad eating habits and so on), then we have certain body features that are controlled by DNA and probably certain human behaviour that is reflected in our children ( "he takes after him/her" ).

Just the recognition that behaviour can be transcended by children makes a strong case that the building up of the human brain is determined by rules that are prescribed by the DNA, a kind of "brain blue-printing", a recipe for how to build a brain through a set of rules.

So, we could create a neural network through very random rules and see what happens, but we could also think of the construction of that network to have followed certain rules that are determined through evolution. This would make a particular network more effective at each generation. It's a big question. Real connections are formed by neurons that just happen to be close by another and I cannot imagine the possibility that a neuron on one side of the brain manages to connect to a neuron at a significant distance.

Maybe the construction of this network is determined by a lower level of emergence, which is determined by smaller elements like DNA and whatever else is there at an organism level. Perhaps our consciousness starts with those minuscule elements?

Or just maybe the growth of the brain is entirely random. We could then consider the possibility that neurons exist somewhere and grow towards another. Then, through Hebb's rule, it might continuously attempt to reverberate and kill those axons between neurons that never lead to reverberation together (thus, have no useful interconnection with one another). Especially in the first four years, these connections (axons) grow like wildfire in a continuous manner. It takes four years for a network of 50 billion neurons to start producing some sensible results. We generally kick-start a network and almost expect it to produce something interesting after five minutes.

It would be very interesting research to find out if this kind of growth/evolution can be jump-started and done in much less of the time through application of a computer cluster (or whether the brain can run on clusters in the first place :).

On content and process

I read a very interesting post just recently regarding the difference of content versus process. Process is basically determining action based on the context and is very much done in the here and now. Content has to do with analysis of concepts and the relationships between them and could be taken as learning experiences. Process can also be learning, but the enhancement of action (reflex) on the perception of content identified in a way. Content itself is deep-rooted knowledge of how a concept might have gotten somewhere or how it might relate to other concepts (in various different possible ways).

Maybe if you don't appreciate the arts, you're a person that highly prefers process (objectives), getting things done or moving from A to B without caring much about the how and where. People that really dig art and content may not be as efficient in getting their things done, but they understand the relations between concepts better and "enjoy the journey" :).

This is an interesting differentation of course. If not achieving your objective causes frustration, than this might also explain why some people feel depressed, frustrated or stressed more than other people. Some are just there for the journey and the pleasure, others always want to be somewhere else just as they got somewhere.

The argument of the person writing the article was that long, continuous exposure to video games and films for example didn't train the content-analyzing capabilities sufficiently. Therefore, training people to only get things done without training them on the pleasure/enjoyment of analyzing the interrelationships and contents of things.

Historically, humans have mostly lived together and generally spent a lot of time interacting with one another, developing and improving interpersonal relationships. The virtual environments are however loaded with objectives "just to make it interesting", so the argument that social environments improve relationships isn't a natural argument. It might just provide an excuse for achieving your objectives.

The article also articulated that the development of the individual (the recognition of who you are yourself, your "self-idea" ) isn't as developed. Or stated in another way, you're not sufficiently self-aware or "individualawared" enough. This puts pressure on the need to make more effort to be recognized as a specific type of person, or projected self-image.

This lack of individuality could then be compensated by collecting status symbols, generally projected symbols of what is considered success by oneself. Those symbols of success are basically material trophies like cars, houses and other things, material things that are thought to add up to one's identity. Sadly to say though, one can never gather enough items for individualization, there's always place for more "self-articulation", which explains the unexhaustive search for new items to conquer.

Cocomo II implementation on Project Dune

As discussed in previous posts, I've been designing an approach for Cocomo II software project estimations. The implementation has already been added to Project Dune on the main branch and is destined for finalization into a new major version of the project, Project Dune 2.0.

The first step in estimation is to determine the project size:

http://gtoonstra.googlepages.com/sizing.png

The next step in this approach is to determine the effort multipliers. Those multipliers increase the time required to develop something linearly.

http://gtoonstra.googlepages.com/effortadjustment.png

The scaling factors in the next tab increase effort exponentially, so if those end up really high, the effort required is soaring as well:

http://gtoonstra.googlepages.com/scalefactors.png

After the factors are supplied, the outcome of Cocomo II is a set of numbers. The most useful numbers at this time are listed here:

http://gtoonstra.googlepages.com/result.png

Notice in the different screenshots how the factor being manipulated is explained in the tooltip. The text in the middle is a reflection of the correct assessment of that factor.

The results show the person-months that are required to develop the project, the nominal person-months (if all scale factors and multipliers would be nominal), and the Time-To-Develop together with the number of staff required to implement the project.

The art of software estimation

I'm reading up even more on the Cocomo II method and interestingly, I like it. Although it's all maths and you can't run companies through formulas of course, the exercise you're put through is the real value.

Consider the following definitions of estimate:

"to judge size, amount, value etc, especially roughly or without measuring"

"to form an idea or judgement of how good etc something is"

"a calculation (eg of the probable cost etc of something)"

So estimating is about forming ideas, judging and calculation. Cocomo II does just that. No estimate method can replace the judgement part, but it can provide the math part.

There are loads of parameters to consider, but within the context of a formula you get an idea what their impact is when you deviate by some values. For example, some parameters only slightly add cost and time, whereas others have a strong exponential factors and an incorrect evaluation of the real value has great consequences.

One of those factors is regarding re-use. For regular projects I worked in, people wouldn't properly go through the practice of estimating the re-usability of some piece of software. Clean interfaces, excellent documentation and little coupling with a used library means it can plugged in within one day or so. But once you plug in a library that has good documentation, probable bugs, but controls your software (like Spring), the effort of integrating and configuring it into your code suddenly becomes a lot larger. Unfamiliarity with the business sector, the library code, other packages and how they work together also adds up linearly to that effort.

There's a great opportunity here for a new opensource tool on the web that supports these kinds of estimates. And it has to be written for the non-cocomo savvy users. Estimating with Cocomo only really makes sense once you are working in teams of 3 or more for 3 or more months. Anything before that is probably better off with a hand-written estimate on a piece of paper, because the size isn't considerable at all. However, be aware that the accuracy very quickly wears off after that and you need to be put through a real practice of estimation. It's really an art form, but nothing mystic, just common sense. And I think anyone will produce incorrect estimates. As long as you understand by how far you could be wrong and supply that value along with the estimate.

For Project Dune, I intend to make something quite easy. The idea is to explain the purpose of the parameters and allow the user to tweak them. Then on a side-bar show a real-time graph with the impact of the changes. And everything in a kind of wizard form.

Estimates are based mostly on code size. And size can be expressed in either Lines Of Code or Functional Points Analysis points. Then you should use a multiplication factor and apply the formula and out comes your cost and timeline.

As said before in the previous post, I wouldn't be so much interested in the actual estimate produced, but much more so in the potential variance. Probably with some more maths behind it, there should be a possibility to show range graphs that indicate the very worst case, the likely bad case, the real estimate, the likely good case and the best case.

The estimate is used as input to project planning. It doesn't yet adjust for serial / parallel work breakdown structures. So the real planning that states "how long" a project takes is still different. You may have people idling or crashing tasks to get things done.

I'm sure there's more stories on estimation at a later time in this blog.

Software Cost Estimating with Cocomo II

I'm reading the book Cocomo II. It's an adjusted cost estimating model from, mostly, Barry Boehm. As you can gather from previous posts, I'm sceptical on these models of process improvement and managing projects through "numbers" and forms. Ideally, we know everything at the start of the project and everyone knows how to do things most efficiently.

If you read the book only literally and pay attention to the math, it won't get you far. The power of the book comes from the interpretation of the ideas behind those numbers and formulas and only then use the numbers anyway, since they're the only factual foothold you'll get in a real-life situation. The rest are fictions of our imagination or our personal recalibrations that are influenced by our own ideas of project management.

First you should remember what the word "estimate" means. It's an expectation of cost, time or effort based on the information known at the time of producing the estimate. One of the strongest and most interesting statements is done in Chapter 1, where it is made clear that Cocomo II doesn't give you drop-dead fixed deadlines or cost statistics (as long as your numbers are correct), but that they provide a guideline or foothold on a potential track for your project that is subject to a potentially large deviation. The size of the deviation is determined by the quality and quantity of information at the outset of the project. So, the more you know beforehand, the more accurate your estimate will be. That is a logical given. Maybe that is also why you should consider reproducing estimates at different stages throughout the project.

However, things go further than cost estimates. In estimates we often make assumptions and tend to think along a positive trail of project execution. That is, we generally like to disregard risks and things that will go wrong and just don't take them into account. Or we think we can squeeze the effort anyway and do more in the same time than initially envisaged. In order to be correct in the estimate, regardless of a boss that won't like what you're telling him, you'll need to also factor in the negative issues into the equation.

So, there must be a number of factors that negatively impact the project outcome as envisaged in the estimates. Think of these factors for a future project, as these increase the potential deviation significantly:

• Incomplete definition of scope at the start of the project.
• Unclear development process or not living up to that process.
• A development team that doesn't communicate well or otherwise faces challenges in its communication.
• Scope creep in future stages of the project.
  

There are some people that understand that certain projects above a certain size or cost have no chance of succeeding. That is because the environmental factors keep changing and due to the size, the communication needs to increase and all other factors. Simply put, you can't reliably estimate on those projects, since there may a deviation of a factor 4-8.

If you think to the completion of the project, it would be very easy to compile an estimate at that stage. You go through the project history, estimate how much is lost for each event (without considering the real numbers) and chances are you'll arrive at a number that is about 90% accurate. But we can only do that because at the completion of a project, we have all the information we need to produce that estimate.

Now think towards the start of the project. What information do we have available and what are risks or issues that we should foresee?

From cost estimation, I think we can learn that reducing project risk and improving the chances of success is increasing the amount of information for successful development of the project. Think of methods like software proto-typing, iterative deployment cycles, showing things early, etc. It's not yet proven that these provide the correct results, because showing things early may also induce a feeling to your client that things may change at any stage in the process.

So, from all of this, I can conclude that the estimate itself is not the most worthwhile thing produced in the cycle, but the accuracy of that estimate is more valuable. How much deviation can we expect overall? And how do we express it? Since we can't reliably come to any estimate at all to initiate a project, what range can we give to project decision makers, our sponsors, so that we can inform them beforehand whether something should be done or not?

Probably, this conclusion should result in a whole new way of software development. Something based on measuring the quality of information, scope and specification available at the outset and a measure of the risk involved if things are progressed based on that little information.

Requirements of a silicon brain: Project Semantique

I've started work on some implementations (research) to elaborate ideas on the implementation of a symbolic network. A good method for me is trying out some implementation, switching to reading, switching to philosophical meanderings and back again. The entire process should feel like some kind of convergence towards a real working implementation (how far away that may be).

In previous posts, I touched base with a couple of requirements that are part of this design:

1. The energy equation must hold, that is, the time and energy it takes for a biological brain should more or less equal the time and energy of a technical silicon implementation
2. It should be parallel in nature, similar to neuron firings (thread initiations) that fire along dendrites and synapses
3. It should be stack-less and not need immense amounts of stack or function unwinds
   
   Some new requirements:
   
   
4. The frequency of introspection is undetermined at this point, or better yet: "unknown". I don't know how often to check any kind of result in the network to come to any kind of conclusion or result. But I reckon that the frequency is tied to the clock-cycle of the main CPU or anything that can be compared to that. So, someone noted in my blog that the frequency of the brain seems 40Hz. That means it might be needed to inspect 40 times a second (and cleanup old entries, leaving room for new?). The idea is to not push too much in the heap for analysis, but clean up the results regularly and continuously work forward, storing the previous results in different rings of memory
   
5. There should not be any "output dendrite" or "return object".
6. The state of the network at any point in time == the output.
7. Previous results should eventually be stored in different rings of memory, which have lower qualities of prominence the further away from the source of processing. Most possibly, results in more remote rings of memory may require re-processing in the brain to become again highly prominent.
   

So where does one start to design this?

I'm looking at "stackless python". It's a modified library of python that allows little tasklets to run that do stuff. Basically it's similar to calling a C function that passes in another function address to execute. The calling function unwinds and the CPU can start executing from the new address.

Python further hides tasklets (to run in the same thread), some kind of "green" thread or micro-thread, since it has a scheduler (that is not pre-emptive, but cooperative).

Check it out here:

http://members.verizon.net/olsongt/stackless/why_stackless.html#the-real-world-is-concurrent

What is the objective?

The objective for now is to load word lists and process text. It's quite a basic process I'm simulating at this point, but that does not matter, I'm mostly interested in seeing if these methods display any kind of emergent intelligent behaviour:

• loading word lists in memory
• Enter 'learning mode' for my symbolic network
• Process 'stories' that I downloaded from the web
• Establish 'connections' between symbols
• Verify connection results
• ... modify algorithm ... modify implementation ... back to 1
• post results
  

Project name: Semantique

On the implementation of humor...

Cool! April Fool's Day. Well, I did not hear a lot of jokes today, luckily, but I guess that others will have been fooled at some point one way or another.

From cognitive science, I am very interested in the analysis of humor... What is humor? I'm not asking how to tell a good joke or what makes a good joke, but on a lower level I'm trying to understand when we find something funny. So how come something is experienced as funny?

I define humor as the deviation of the most logical / expected path of the change of the context (your expectation) towards something that you didn't immediately see coming as part of the analysis of the development of the story. Well, that's a joke anyway, otherwise if you did expect it, it wouldn't be all that funny. The best jokes and joke tellers keep you from the other logical, explainable path long enough until the punch line, where the actual context all becomes clear.

All well and good... Star Trek seems to suggest that Data could not understand humor, as if humor in its essence would be pure human emotion. Since machines don't by default have access to emotional responses (if emotion is the driving force of our life, in the sense that it is at the basis of our decisions in the morning to get up and start doing something), then Star Trek would assert that Data couldn't laugh at some joke because he didn't have access to an emotional organ or simulation of such.

I'm not sure of that assertion made in the series. I think humor isn't as much emotional, but more a trigger (spike?) in your brain that brings forward an emotional reaction (laughter). That little difference is very large. Scientists performed experiments on "aha" moments, those quick moments that you have solved a puzzle and can complete it in its entirety. Those "aha" moments were accompanied by huge spikes of brain activity for a very short time, after which the context of some problem should be entirely clear.

I do imagine humor (specifically for now) thus to be an emotional reaction to a relatively simple discovery in the brain that the context and path we've been led to believe (our expectation) is not the real path we should have taken to develop the context (chain of symbols) of some story. And by "shifting" this context the right way due to more information becoming available (the punch line), we feel a response to this "aha" moment when the brain solves it. Also, if you concentrate, you can rather easily suppress the urge to laugh in a great extent. (does that suggest that laughter and humor are quite conscious processes?).

(could you say then that the closer the expectation is to the actually developed context, it makes the joke funnier or vice-versa, the farther away using the same words makes it funnier?).

So, anyway, that means that there might be ways to detect humor by software as well, provided there is software that can develop expectations and interpret contexts the same way our brain can.

Therefore, Data won't probably actually laugh in the same way that we humans do (since biologically we react to that aha moment), but probably it is possible to detect if something is humorous by analyzing the contextual difference in two different snapshots of the context and then send the appropriate signals to react to it. Of course... Data is most likely not culturally apt as he does lack real biologically induced emotions, so he may very well laugh inappropriately in contexts that are culturally sensitive (imagine!). But that is another story.

P=NP?

A new post on Silvio Meira's blog today. Apparently, one of the math professors over at Pernambuco in Brazil is going to post the proof that P=NP.

http://silviomeira.blog.terra.com.br/p_np_sera

Can't wait to see the result. It would have very large consequences for everything and computers will definitely be much, much, much more helpful than they are today. Stay tuned for that story.

I'm following it as well and see what happens!

Three Levels of Analysis

One interesting book of Pinker on Cognitive Science discusses the three levels of analysis, it is also on Wikipedia (since it belongs to the field of CS). Anyway, these three levels represent three layers in the OSI model of computing. The physical layer, the algorithmic layer and the computational layer. The algorithmic layer would be compared to the Operating System and the computational layer to the behaviour (which would be applications making use of the infrastructure).

Of course, when mapping these levels to biological counterparts, they map to neurons and so on.

The interesting statement is that it is not only required that you understand the workings of one layer individually (to be able to make sense of something), but you'd need to understand the interaction between the levels of analysis in order to work out how something works.

There are some potentially large pitfalls there. We make assumptions that a neuron has a certain use and can never have more than one use at the same time. I have no information at this time to make any further statements on that though.

One of the questions in the books is what intelligence really is, and it is then purported as some kind of computational machine. Computation is often associated with algorithms, formula and maths.

I feel happy about people saying that mathematics and algorithms somehow describe the world (or can be good means to communicate difficult issues or describe some kind of event), but I don't think it's a good idea to turn this upside down and then state that the world is mathematical in the first place. It's way more dynamic then that. It's maths that describe the world (and then somewhat poorly, as some kind of approximation).

Although very complex things can be described with a sequence and combination of many algorithms together, this presents enormous problems. First of all is that it makes the world deterministic (although incredibly complex with algorithms connected and influencing together) and second is that in order to come to a full implementation, you'll need to understand absolutely everything and model it in some kind of algorithm. That sounds pretty much impossible and a 'dead path'. I think AI needs something more dynamic than that.

There was a lot of enthusiasm on Neural Networks. I've used them too and I like how they operate, but in the end a network with its factors is only useful for one single task. Another task cannot be handled by it, unless it is retrained. So those are very limited as well, plus that I find the speed and method of the human brain for learning immense. Another limitation of NN that I find is that they require inputs and outputs and have a fixed composition for the problem at hand (a number of layers and a fixed number of neurons and synapses inbetween).

So, NN's are also deterministic and limited to its purpose. What and how should something be designed so that it can seem to learn from something and also start reasoning with its knowledge? Reasoning in this context to be interpreted as a very wide term. I explicitly did not use the term compute to make a difference. A Design is often very limited, limited to its area of effectiveness and to the knowledge at hand at the time. The design is limited to our disability to design indetermine things, items that may produce inconsistent and unexpected results.

When we design something, we do that with a purpose in mind. If this is the input, then the result should be that. But this also limits the design to not be capable of doing anything more or different from what it was designed to do.

How difficult is it then, with our minds focused on algorithmic constructs (consistent results), if we are just now trying to work out a design for something that may produce inconsistent and indeterminate results? It's the toughest job ever!

Some more about symbolic networks

I've created some images to explain the ideas of the previous post. I think those ideas are very promising, since they indicate they could comply with the requirements/statements of previous posts (I can't academically make this statement until I have scientific proof of this, so it's a hypothesis):

• The energy equation between a biological network and a mechanical network should more or less hold, within a certain range and some research should be made if there needs/should be a factor in this equation.
  
• There shouldn't be an unroll to the functions that are called as part of the symbolic network
  
• There should not be an expectation of an output state/measurement (the network *is* the state and is always in modification)

The following is a PET scan that shows brain activity. Think of this as a screenshot at a certain point in time, when the network is processing some kind of thought:

You can see some areas of the network are totally unused, whilst others display high states of activity. Of course, it is very important to assess the brilliance factor and brilliance degradation/fallout (time it takes to decrease the brilliance) within the context of this picture.

The brilliance is basically activation of neighboring nodes. So thinking about one concept can also easily trigger other concepts. The "thread" of a certain context would basically guide the correct activation path.

I imagine a kind of network of symbols that are interconnected as the following picture:

The "kind-of" association is not shown here, because I'm not sure it really matters at this point. The "kind-of" assocation in itself can also be associated with a concept, that is, the "kind-of" can be an ellipse itself. So there is some loss of information in the above diagram, but that loss is not being considered at this time.

You can see that concepts are shared between other concepts to form a very complicated mesh network. It's no longer ordered in layers. If you consider the strength of an association (how strongly you associate something with something else) as the line that is inbetween it, then I could ask you: "What do you think about when I mention exhaust gas?". Then your response could be car or bus. The lines thus represent associations between concepts.

Wheels are known by both the concept car and bus. Also notice that this network is very simple. As soon as you gain expert knowledge in a topic, this network will eventually split up into sub-topics with expert knowledges about specific kinds of wheels and specific kinds of buses and specific kinds of cars and how they relate to one another. Generally, we distinguish my car and other cars, which is one example of a topic split. This statement of expert knowledge is derived from my little nephew looking at his book. For him, a motor bike, a bus, a cabriolet, a vw and things that look the same are all cars at this point in time. Later on, he'll recognize the differences and store that in memory (which is an interesting statement to make, as it indicates that this network is both a logical representation and association, but also memory).

The connections in this kind of symbol network can still be compared to dendrites and synapses. The strength of an association of one concept with another is exactly that.

Now, if you consider that you are reading a story and you have certain associations, you can also imagine that these concepts "fire" and are added to a certain list of recently activated symbols. Those symbols together form part of the story and the strength of their activation (through the synapse strength, their associations with other topics and a host of factors, basically what the network has learned) will in different contexts slightly change the way how the gist of that story is remembered.

If you store the gist of this list (produced by a certain paragraph or document), it should be possible to compare this with other gists through some clever mathematical functions, so that gists of one document can be compared with others. Gists are also methods of reducing storage details and storing it in a much compressed form.

Consider the final picture in this post:

It shows a simple diagram of, for example, what could be a very short children's story (well, we're discussing this text at that level basically). Dad goes home in his car and enters the house. He sits on the couch and watches the tele. If you remove the verbs of these statements, you'll end up with a small network of symbols that have some relation to one another. I feel hesitant to jot down the relationships between them in this network of symbols. I'd rather add some layer on top of these symbols that manipulate the path that a certain story or context takes. So, the concepts are always somehow related, but the thread of a story eventually determines how the concepts really relate to one another. Therefore, the thread manipulates the symbolic network in different ways.

So... what about design for an implementation? In game design, even when it was still 2D, the designers already started with large lists of events and lists of nodes for path finding for example. Between each frame, these lists were rebuilt and re-used to update the AI or action. Those design patterns should be reusable in this context:

• Start with reducing a text to its nouns only
• Process the nouns one by one
• For each noun:
• Reduce the activation factor of current concepts in the activation list
  
• Apply the synapse factor to the current noun
• Add concept to the activation list
• With a reduced activation factor by synapse, add related concepts that are connected to the currently processed concept to the list as well
  
• Get an inventory of the highest activated concepts in the list
• Store the gist list to describe the text
  

Obviously, the thing that is missing from the above is the intention, the thread. So a text that describes a guy in India getting of a bus to his house may equate to the same gist of a bus in San Francisco that happened to drive past an Indian restaurant.

So motivation and thread of a story is something entirely different from its concepts. Should this be part of the network?? In a way, I think it should be possible to think of it as a layered network above the symbolic network, a different kind of representation with links to the other network to describe actions and objects that are acted upon.

It's the state of the network, stupid!

Back to the discussion on Artificial Intelligence and Neural Networks. This blog hosted some meanderings and thoughts on AI and how this relates to NN. I read books from Steven Pinker and others regarding NN's and I understand how these work from a helicopter view (I even implemented some). I then analyzed that the perspective of NN's on AI is probably horribly wrong, if you compare biological networks against mechanical ones.

Computers were from the very start designed to work on input and then produce output. Output is a very binary-like state. It's either this or that, it's a range of numbers and the output is generally an exact number. There's not really a way to represent an answer that represents two (slightly) different states in one answer by itself.

This morning, I woke up and considered that this general approach that was taken as part of AI way back is probably wrong. Even though computers are supposed to produce output that can only be interpreted in one single way, the "output" of human brains doesn't really exist as output per se. I'm more thinking of answer and thought as a kind of "state of the network" at some point in time. The frequency of thought is given by the "frequency" of the network, although this seems a very weird term to use for biological networks. It's probably totally independent.

If you look at CAT-scans though, you'll see something interesting. Not all neurons are active at all points in time (very contrary to mechanical networks, which generally have all their nodes and parts connected, so that turning one part will turn and impact another). And the granularity of our analysis on the human brain is not at neuron level, but at the level where we see a general number of neurons receiving activity. So if one neuron A next to neuron B is fired, only A would be active, but B would not be assessed.

So it's like regions of interconnected neurons are active at one sweep, not the entire network. And there's no output like a machine, only a list of active and recently active neurons. Every sweep, the active list is modified and moved back into a ring of memory.

If we reconsider neurons as nodes in a network and replace them with symbols instead, we can probably come close to a logical representation of a thought network. So, a neuron by itself doesn't represent anything, but impacts something physically. A symbol is highly representative of something, but doesn't necessarily impact anything, it is only connected to other symbols.

The symbolic network is then like a neural network, only it works with nouns, with symbols. The symbolic network allows an infinite number of nodes to be added, as long as there exists a process to interconnect symbols as soon as there is a relation to be determined between them.

Now, imagine what happens and assume this symbolic network is properly connected. When you mention a word car, or the smell of exhaust gas, or a picture of a car, those symbols are activated. The joint activation of car, exhaust gas and the picture should activate a symbol of car (without annotation) as a concept, so that the network understands that car is being discussed.

If you now introduce a text with nouns and verbs and assuming the computer has grammatical analysis capabilities, you can process this text within the symbolic network and at the end of some paragraph / passage of text, the network has a certain state of activity. Some regions are highlighted and other regions are black. If you'd keep a list of symbols that are activated, then you could store that list (region) as a representation of the text.

So, the objective is not to store the text word for word, but to store the associations and context of the paragraph. Mentioning the words in a search term would probably produce the text again and the more aligned with the paragraph it is, the more likely it is to be found.

Memory is also important. There are actually different rings of memory in this model (short-term and long-term are too generic). Reading a passage would store the gist of that passage into a different symbol. The gist are basically those nodes that had the highest activation of a certain paragraph after the cycle is completed. So storing the gist of one paragraph with another may develop a description of a document that is highly descriptive. It's not necessarily the word that is mentioned, it's the concept and the relation to other symbols. It's possible that a symbol is highly activated that was not explicitly mentioned in the text.

The symbolic network is the representation of nouns of our language, but verbs are the activities, the manipulations in our mind of those symbols. It seems then that within the context of this blog post, the verbs correspond to real intelligence (which is not described in this post yet). The nouns are just perceptions and mapping them to symbols. Real thought, the one that creates (is creative) and can reproduce and come to certain conclusions is a totally different matter. That sounds like real artificial intelligence.

Project Estimation

There are some interesting discussions taking place on project estimation. Project Dune 1.4.2 came out yesterday, where you can register your estimates per activity per scope statement. So that's basically "ballpark" level of detail.

There are some new requirements that are coming up that look really interesting. PM's really seem to (want to) use some kind of math magic to get to certain numbers and then tweak them.

So, what really matters in a project? What are factors that, if they change, significantly affect the timeline of a project?

I've already mentioned that the level of communication is a huge issue. So, if your team increases, you'll increase contention within the team, the need for communication (which loses time) and so on. Especially at the start, where the PM actually intended to move fastest. So the rest of the trajectory people are generally told to "get on with it", leading to unsatisfiable levels of communication and lower standards of quality.

Oh well! :)... Expect some more notices in the future with regards to this topic.

Screenshots and features Project Dune

I took some time to update the projectdune wiki with a list of features currently supported. That should make things a lot clearer. See the screenshots for further details.

Supported features

Cold!

Err... yeah, it's been cold in Holland for the last two days. Icy, salted roads and scratching ice from car windows. It'll improve very soon though.

I worked on Project Dune over the weekend and it's now getting there. One more bit of functionality to create the reporting ability from estimate to timesheet. That'll certainly mark a milestone. From there onwards as they say. I'm trying to take things easy now and just improve on existing things. Maybe some refactorings here and there.

There's still a new module in the pipeline for test case management. So you can write a document, the test plan and then refer the individual bits with the test results. The same things should probably be done for issues.

Identity on the Net

Project Dune has a feature implementation pending with regards to logins through OpenID's. When you surf the net nowadays, every site you go to requires you to put your details in for registration. The objective of registration is to be able to verify the same details at a later time. The details are verified by a secret that you are telling in the process of registration (your password), which is stored securely at the site and of which both parties hope is a secret that is never shared with any other party.

Your registration however is further verified by adding a key that you only own: your email address. The registration site creates a temporary link with a complex url that is very hard to guess. You move to the url and thereby activate your account that you requested to set up earlier. From here on, you don't need to keep checking your email to be able to log in, you use the shared secret.

What most sites care about is authentication. Authentication is the process of verifying that someone who identified himself with some token at some point (your name or any other credential) is the same person that comes back another time and not somebody else. This is essentially what authentication means for sites that allow "semi" anonymous access (you register with your details, but the registration site never actually verifies your real identity. So what they check is not that you are who you say you are, but that you are the same person that made the initial registration).

OpenID attempts to simplify this a bit further. Rather than "registering" in the same method for each site that you may be on, you'll continue to see new sites popping up that claim to manage your identity online. Basically, this means a split of the registration process from the authentication process.

Remember how I said that many public sites don't care about who, in the context of real identity, is making a registration? They only care if they can reliably assume that it is the same person that actually registered before. So another site could authenticate on the site's behalf and then have the user call back to the originating site with the result of the authentication.

The authentication relies on the fact that the user now claims to own a url. For example, if I want to log on with OpenID on a forum site, I could claim I own:

http://gtoonstra.blogspot.openid.com.

Then the forum site tells me: "oh yeah? proof it!". So they redirect my browser to an authentication provider site "gtoonstra.blogspot.openid.com". The forum site passes in a return URL that is used to pass my browser back after the authentication claim. First, I need to fill in my secret that I have with my authentication provider. If I succeed, it is assumed I correctly own the URL. If it fails, my claim is invalid.

In both cases, I get redirected to the forum site. The forum site then analyzes the details and shows the results as they have programmed. There's another process there related with cookies and multiple redirects. So if you have already logged on with your OpenID, a cookie is stored in your session. Then for each site that you visit, you may be requested how long you wish to keep the cookie and the session. As you get redirected from one site to the other, the sites still redirect you to your authentication provider inbetween to re-verify the details. This process is called "single sign on".

Project Dune thus will start to use OpenID. The objective is to modify the project code to allow anonymous browsing for "the public". So you won't have the same abilities as users created through the system, but you can still browse issues and participate in those things that the administrators have indicated as allowable.

Now, consider the options here. Project Dune for example contains functionality for inspections. This means you could configure the software to allow people to participate in inspections of source code. How cool would it be to have anonymous users inspect your code online for certain changes and make remarks? The current method is mostly by sending email, but this way you just login, go to the source, open it up, add your comments and move in. You are identified by your OpenID, so you will in some way be traceable.

Project Dune release 1.3.5

I've worked a bit this weekend on a new installation method for Project Dune. Some software written in PHP is earmarked by the ability to just extract it to some web directory under Apache and run it.

Project Dune now also gains that ability. It's quite unique for Java packages, since mostly these are very manual processes. I spent quite some time to develop a web based installer. It's a war package that goes to Tomcat, where it will configure the project before it puts it in the final place for installation.

Have a look over at SourceForge and let me know your thoughts over at the forums.

Internet and updates to projects

I finally got Internet now. It took three months to get it all done. Alice Internet is crap. I requested it 25th October. Then 10 days later received a letter/email that my service was canceled. No reason given. I called and they said it was the register of connections where my request wasn't listed. So they tried again. Some days passed and nothing happened. I called to be sure and they said they'd restart. Went on holiday for 3 weeks. Came back. Absolutely nothing, as if the request just died. They said they'd try again and I just gave up there. So I went with KPN Comfort or something.

They said they'd connect in 2 weeks, but this became 5 in total. After 2/3 weeks, you get a call to confirm a repair man to come around. Actually, all he does is check the signal on the box and if that's allright, he leaves. So not really any use there. The crap is that only after they ship the box with the modem, so you just wait a stupid amount of time before you are connected.

Anyway, it's done now and working. 650 Mb/s effective downstream. Some 12'th of that upstream. It's stable and has good service.

I've updated Project Dune at SF now and XssProtect at Google Code. Later on, I'll have to redeploy Project Dune because it requires Java6 and has a couple of fixes here and there. Then I'll need to figure out how to get the project more popular. Probably relates to creating complete deployments or shell scripts that can do this. I'm not tooo happy about izpack, because you can't install it on a headless machine. What's the best method to create installs for software on Linux?

Happenings

Still no internet yet. I ordered this end of December and it is taking 4 weeks instead of the promised 2 weeks. At least the connection is confirmed working. As soon as I get the modem, I can start putting things in place.

Yesterday I picked up my car, a Honda Civic Hybrid. You'll see a lot of these in Holland because of two tax cuts (a steep discount on the catalogue value when you buy and other discounts on a monthly basis that is the reduction on lease price). Cars are heavily taxed in Holland nowadays. When you buy one, you may actually pay some 40% of taxes on top of the selling value. What they don't really tell you is that when the taxes are determined on the monthly basis, they don't use the value "out-of-factory", but use the value with the taxes applied. So you pay tax-on-tax basically. Anyway, I don't have too much of that.

I'll be doing a CISSP course soon, the book arrived. It's huge, 1100 pages on general computer security. That includes everything basically, including physical security and so on. That's a good investment to do.

There's a team that does business development at Sogeti with regards to security. I'm part of the architects that find entry-points there. I'll be more of a focus from IT risk management / security considerations and how this applies to architecture. This also includes reading books on SOX, reading general books on IT security management, the CISSP book and finally some insights from the insurance industry (risk management) for some theories and models I am developing. Eventually it won't just be theory of course, I'll need to convert it into practical methods and examples and perhaps test it against previous cases to see if it could have changed decisions.

I'm driving around the country today with family. Tomorrow just an easy day.

Imagination == (re)Design?

The BBC (Prime) is running a very interesting series since yesterday. It's called "The Human Mind" and explains a lot of the issues that I was talking about in this blog over the last few weeks.

Whereas academic institutions mostly work within certain academic constraints, you need to come up with proof of certain findings or intuitions and validate your imagined model against current knowledge. Sometimes you invalidate existing knowledge, most of the times it more or less falls into place or enriches it.

Some great inventions or innovations come from people that had a desire to do certain things. The invention of the airplane, the invention of electricity. But electricity probably wasn't invented within the big picture of having electricity at every house. And probably neither was the telephone. At some point, it's just pure curiosity that drives a great invention, which is just used and exploited later (MS invented Web2.0 through the XMLHttpRequest object, but only some years later was it used widely in browsers. Now many sites popped up that constantly use it).

Can one design without imagination?

Imagination being visualization, proto-typing in your head. Design is then not just coming up with the idea, but making it clearer and clearer until you have it ready to be written on paper. Then the skills of course to visualize it properly on paper or in digital form.

If one were to re-design a very complex piece of work, say, the human brain, then you can't really do that unless you have an imagined idea of how it should work. Most of previous work available deals with Artificial Intelligence, which you could also call "pattern recognition".

In some other comments, I stated that the brain may be this big pattern recognition machine, but it still doesn't explain the kind of signal that flows through the brain. But maybe the signal itself is not that important (that is, there might not be meaning in the signal), but maybe the triggering of that signal through its amplitude and its length is.

Most of science is finite. This is to give us certain limitations and scope of reasoning. Wherever we encounter something that is not finite enough, we have quite severe difficulty in toying with it or imagining the little things that it is composed of. So breaking things apart in smaller chunks, whether functional or through other categorizations, helps to solve the eventual problem. Subdividing it into chunks that different experts treat differently also helps (like building an airplane).

As with previous posts, I think linear memory is the wrong kind of model for artificial intelligent machines. Maybe radial memory would work or spherical memory, where symbols are aligned close to one another and where travelling away from the center might induce some kind of special meaning (so that travelling "through" that memory from one part to another somehow might be steered by thought).

So, that is the essence of today. In order to come up with some kind of redesign in computers that at some point in time might match the human brain (or part of it), I'll comply with the following rules:

I should first imagine how it works down to a certain level of detail.

The energy equation should more or less hold.

(that is, the computer should not consume more energy than the human brain on a daily basis).