Genetic neighbors are like k-medoids

Well, I've found some new info on the Internet whilst trying to refine my algorithms here. I'm still in the process of aligning the proper techniques to be able to run a prediction through. The next one isn't optimal still, but should carry along a couple of good improvements.

The technique I "discovered" actually already existed. That's very common, so no problem there. The application and context in which I'm using it though is very different.

In k-medoids and knn, the trick is to select a centroid or "default" that will act as the center of a determined neighborhood. In KNN, generally parameters are averaged out, as such developing an idea of a class of a certain kind that you either match or not. In k-medoids, the trick is to find out which member is the best representative of the entire set (so no averaging there). This is done by finding out the cost of swapping the center to another member.

This thing above is a measure for similarity, otherwise known as "silhouette" in k-medoids. The silhouette provides information how similar a certain data point is in comparison to its default (in my case, the gene). a(x) is the similarity of the data point to the currently allocated gene, b(x) is the similarity of the data point to the next-in-line highest similar cluster. Thus, if there is a very good match with A, but a not so good match with any other cluster, then the silhouette will tend towards 1. If the other cluster is better (doesn't happen in my case), it tends towards -1. 0 is indifferent between A and B.

Now, the trick is to initialize the clusters to some values. I'm still considering how to do this to a better ability, but I'm also considering it may not matter too much in the beginning. It's optimization. Then, set the ratings for each gene to the exact ratings of the training data of the "default" member that was selected. This probably works better when you select defaults that have a high vote count.

Then, run an algorithm that determines similarity. I'm using ((1/rmse)/sumRmse) as a dirty trick for now. It seems to work well. Then record the similarity for each cluster on each customer (so we are classifying customers here for now). Movies can also be clustered in similar ways, likely. And in that case, one could determine biases between user and movie clusters or even use a different matrix factorization module to svd between those as a "global average". Possibly the mf-values for each user / movie become lower in the second layer and hopefully, through the help of clustering, noise is reduced and the mf values for each individual user become more meaningful and less of an average.

What I'm actually doing after allocating users into clusters / genes, is that I'm using that information to complete the matrix. Not user X <-> movie Y as one big matrix, but gene X <-> movie Y. If the users are really more similar, then the error in movie average hopefully goes down if the clusters are properly populated. The movie average is then used as a focal point for further calculations.

You obviously can't use "average" values any longer, because that's already set by the global movie average, unbiased and over a single gene you could say, the entire population. Thus, we need a new, external algorithm that is able to provide those numbers. Global effects could do for now, or perhaps a matrix factorization method.

The "silhouette" calculated earlier for each user then becomes useful. If the silhouette is a measure for how "powerful" a certain user falls into a gene, then we can set a threshold which users actually contribute to that gene and which don't. If you imagine a borderline between two clusters and a lot of users on that "front", then if one doesn't filter those out, those users would pull the clusters closer together, causing frequent swaps from one cluster to the other, making it less stable (I suppose, this isn't lab tested :).

In initial tests, I came up with 10 clusters and that worked well. Calculation time then is about 20 secs or so per cycle. After 10 cycles, there's just noise generating differences, but no apparent useful changes in rmse. This was before my thresholding algorithm.

After thresholding, the rmse drops quite quickly and also stays rather fixed after about 8 cycles. This may be good or bad, I don't know. I probably need some further tests to see what happens after 64 cycles... Does it then become worse? better?

The genes are calculated with a rather discrete formula, there is no learning involved there. If there is actual training data available for a gene, that is used for averaging out, but only if the threshold for that user in the gene is high enough (it's a measure of representation of that user for that gene). Choosing an incorrect threshold kills your changes and flexbility, too low and there's instability and continuous change.

The gene contains floats, and these floats are "averages", and there are as many floats as there are movies. If any real rates were found for a movie in the gene, then those are used. If no ratings were found for a movie, then the customers within that cluster are used to make predictions (if they have high enough threshold), and those predictions are averaged.

Using this method and selecting k=24 or so, I see the rmse going down to 0.957677 or so after 10 cycles. But the changes and improvements quickly become entirely level after that. Increasing the clusters significantly increases computation time. k=10 is reasonable, k=32 is already pretty gruesome to wait for.

There is a possibility that the initiation of the clusters is horribly inaccurate and incomplete. So it's possible it needs a couple of training cycles to get to a state that it is happy with. I'm not yet sure about the movement and progress of the numbers in this case.

I'm now trying to get better results by combining this method with matrix factorization in specific groups and between groups. Thinking about it... splitting up very large results into groups that are better similar towards another as compared to the global set, it makes sense to investigate more a hierarchical ordering of movies and customers and calculate between those groups as well. However, care must be taken not to make the groups too small or too large. Too large doesn't make the group distinct enough, too small doesn't generate the momentum that the numbers need to point in the right direction :).

Before, I tried to apply the similarity rating to a certain cluster in calculations, but it is woefully inaccurate. It's not really any use taking 20% of one cluster and 80% of another, the results are mediocre. What is much more telling however is the silhouette rating. That rating is much more useful for a variety of purposes, since it is a measure how deeply allocated a user is in a certain gene.

Oh... and today my new memory arrived. Super duper 4GB to run multiple instances next to one another. Although I don't have a quad yet, so it can only effectively be 2. And beyond what I mentioned in this post, there are a multitude of other improvements I am planning before my next submission.

New algorithm: Genetic neighbors

I'm developing a new algorithm that I'm just testing now. I call it "genetic neighbors". It's for use in the netflix prize.

The challenge in the netflix prize is to populate a very large matrix based on 1.5-2% of given values in that matrix. Thus, about 98-98.5% of values need to be interpolated. Some people have resorted to the use of pearson correlation between users for k-nearest neighbor calculation (which in my view is misguided, since it gives a relative correlation between two users, which outside that context does not mean much). This approach attempts to create an absolute ground of reference for correlation calculations.

I was pointed to a link by a friend of mine from (now) the U.S., where it mentioned something about genetic link calculation. This approach basically assumes a set of movie ratings as a type of gene and then tries to match users on that. The neat thing is also that a user need not match a 100%, a user can match 80% with one gene and 50% with another.

The match of the user also has an effect on the formation of that gene, thus, each gene mutates based on the input of different users, according to their match with that gene. The idea is also inspired by incremental "knn", but works very differently, since it's not a relation between users, but a similarity / clustering on an absolute term within this model (the gene). The gene is considered complete in the sense that it has a prediction for each movie within the gene, it is possible to match other users against it wherever they are from. With only very little information, a user can belong to a number of different genes with different weights, such that each gene contributes to the final prediction weighted by its correlation with the user based on historical information. Mistakes will of course be made there, but the more information becomes available, it is expected that predictions become more precise.

Also, I'm experimenting with how this algorithm learns. If a user doesn't have a strong profile (relatively little ratings), then it should not contribute that much to a gene. It's also another challenge (yet again!) to find the correct parameters. I'm also relying on gradient descent for learning the genes.

The difference in this program is also the organization of the rating information. Most programs probably use a type of replicated table in the application in the form of a struct and an array. In this model however, the ratings are allocated under the user structure, which greatly improves the speed in gene calculation and genetic training and also reduces memory requirements.

Doing this cluster analysis, it's also clear to see that the users don't really fit that easily into a cluster. They're sharing some parts of one gene with parts of another gene. Setting up a gene for each individual user would be ideal, but then there wouldn't be any re-use of cluster-specific knowledge of other users.

Something to watch: FACETS

FACETS is probably one of the most interesting research projects that's taking place right now. Well, together with the LHC :). Previously, I posted on the architecture of mind. That post resembles the envisioning of a totally different kind of hardware as it would apply to (re)modeling the human brain. Well, the people at heidelberg uni are now doing this. They're constructing hardware that uses a radically different design than micro-processors. From their site:

By creating specialized digital hardware processors it might be possible to gain an advantage over microprocessor-based systems. Still, it is unlikely that this will be more than an order of magnitude, since they are based on the same technology as microprocessors: The neural circuits remain to be realised with numerical solutions of differential equations. The biggest problem lies in the fundamentals of Moore's law itself: the scaling of process technology. In the current semiconductor roadmap the progress has already slowed down. The transistor density of high-performance microprocessors is likely to increase only by a factor of 25 from 2004 to 2018. A power consumption of 300 Watts is predicted for such a hypothetical chip while the on-chip operating frequency will be in the 50 GHz range.

I've blogged before on consciousness and how this might relate to computers, and what a design could look like. And some more. And more.

Changing the hardware is extremely important and much more likely to become successful.

A large problem is still the coordination between things and the problems related to associative memory. That is, when we say "cat", we instantly recall associations related to the word, strongest first (black? kitten? dead mouse?).

Most explicit knowledge systems scan their entire memory base, or have otherwise explicitly defined boundaries around knowledge to hierarchically exclude certain pools of knowledge programmatically. Thus, a key pointing to some piece of information is not recognized as such and doesn't cause a specific part of memory to highlight. It's requiring a sweep of memory to see with which asset of knowledge it's associated.

In order to be successful in the future, I think it's necessary to find ways to prevent that, to directly find some location/pool/hierarchy where something is probably located, such that it finds a match or has the ability to locate it together with other like members.

Early morning

early morning
Originally uploaded by gtoonstra

Last week I purchased a digital SLR camera for starting to make some proper photo's. I got up early today to see how I'd do, how the camera performs and try to make some good shots. Here's one of my favs, even though the sky is horribly overlit. It's a little bit eery, somewhat mysterious. Sharpness is quite ok, could have been better with a tripod. Good division of fore, middle and background and everything.

More netflix...

In the calculation run tonight, I ran the entire algorithm with a couple of different factors. I did not manage to significantly improve my ranking, but I did manage to get closer to the probe rating by training over the entire data-set, including the probe. I'm now researching at which point data overfitting starts to occur. I reckon it is related to the learning rate and I reckon it is about after ( 50 cycles + current-feature-num ).

If you only react to the probe rmse improvements to determine to switch to the next feature, this may never become apparent. I've run a wide number of different configurations, always using probe rmse as early stopping method (once it stops improving), but now I reckon that it may be too late. I'm now looking at early stopping once the hill has been reached, but using a simple calculation (as above). I'll probably post about those results in the future.

I've noticed that as time progresses, the point of highest improvement moves backwards and you need more epochs to get there. Thus, the first feature shows improvement immediately after the first epoch, but the 10th for example only shows improvement after 16-20 epochs, also "stopping" 16-20 epochs later than the first.

I've tried squeezing out as much rmse improvement as possible by using different parameters and global effect reductions, but this results in a flat line of no more improvement after about 32 features. Actually, applying the movie/user bias global effect worsens the result significantly, so I've turned that off. Instead, I'm relying on the bias training together with the features for the moment.

Using K = 0.03f or K = 0.01f also doesn't have any effect in the long run. Actually, the weird thing is that the probe rmse for the first feature isn't really affected by it at all. Also, in the long run I'm seeing the same numbers come back.

I've also tried gradually modifying K and lrate and neither did that have any effect. Note that in all these experiments, the probe rmse is the leading thing to determine to continue or not. I used to rely on a standard MAX_EPOCH setting that switched to the next feature, where I did get differences. The following sketch is instrumental in my explanation:


The graph shows how more or less the improvements build up and decay. The x-axis shows the epochs, the y is the probe rmse improvement. The vertical dashed line is the cut-off point where training stops. Thus, it shows that whenever training is stopped at the cut-off point, it is irrelevant what the parameters are, since the area doesn't seem to change much. I should have drawn the smaller bell-curve higher, so that the surface area under it makes it clear they have the same gain in total (which in my setup now is the case).

The learning rate squeezes the bell-curve together (allowing earlier stopping), but it also makes the total result less accurate (since overshoot may occur). The K has a similar effect to learning as it holds back (regularizes) the learning. But overall it should not matter too much.

So my approach now is to find out at which point before the cut-off it is best to stop learning and move on to the next feature. My guess is that over-fitting starts occurring right after the training has passed the hill. By aligning my parameters, I'm going to try to put that hill on a predictable point, such that I can simply set a maximum epoch and then move on to the next.

Al go rhythms

Mr. Larry Freeman, see link, is a software engineer from Fremont, who has taken the time to explain the "global effects removal" technique sometimes used by contenders in the Netflix prize and reflects on a number of mathematical issues. It's quite interesting for non-academic people, since it's bite-sized pieces.

I'm already grateful for explaining and demonstrating the global effect removal technique. I've tried putting it into my implementation, but sadly could not significantly improve my ratings. As you may guess, I'm using RSVD together with bias determination á la Funk and Paterek. I'm not using any blending whatsoever.

One of the reasons it doesn't work so well is probably that my implementation already covers biases by training biases along with svd factors in the first phase. Thus, the benefit of it is greatly reduced. I do notice however that using the technique requires my factors to be severely changed. I had lambda=0.05f as suggested by Paterek for calculating the biases, but now that these have been factored out, I reckon they have to be a lot higher to prevent over-fitting.

From the first 3 iterations in the svd algorithm, I can tell how things will turn out towards the end, unless my parameters are really wrong. I've seen a very good first rate at 0.951 (lower than the netflix start average ), which only ended up in a disappointing end rate of 0.9120 (higher than my ultimate post of 0.9114).

Thing that's interesting to research is to factor in global effects into the training phase. I imagine that I'm getting better results with biases, as opposed to global effects pre-processing. That's probably because all global effects are global averages, whereas some kind of gradient descent training may be able to discern the real features and the way they influence (or seem to influence) specific people. Those features may be different from the ones identified for global features, but one thing I'd like to definitely do is use the date more. Consider ratings made on a certain day, find out how much they differed and then make predictions on the change of the date.

This is not related to the production date of the movie versus the date of rating (I think that's very difficult and probably there's no causal relationship between them). One of the tricks in this netflix competition is to find causal relationships, as they make huge differences in the outcome. (in another way, you could say that if a business rule could be developed based on a certain observation that's mostly true, then that's great discovery!).

Improving the position on the leaderboard (up to 884 from 1138)

I've been busy with some algorithm refinements after reading papers on the netflix prize. I've gotten some very gradual results, but tomorrow should come up with some very good improvements after taking out a serious bug :).

It's quite a difficult problem to understand, but with more runs and perceptions of how the runs and data behaves, you get a feel for what to do. My advice to new starters is not to tie down too much to your algorithms. Fiddling with parameters is not going to significantly improve your results. Find ways to understand the data, the crux of the problem is that your algorithm is using a very sparse matrix as a basis to predict values in those cells where no value is present. Since the matrix is only 1.5% or so full, that's not a lot of data to start from.

I'm measuring performance mostly against the probe, not against the test set. The test set is just used to train the features.

So, the difficulty of applying SVD to the netflix prize is that the matrix is extremely sparse, generating loads of inaccuracies, and also that through having such a little amount of values available, the elasticity between values doesn't develop. With elasticity, I mean that you can perceive the original matrix row by row, as similarities between movies, but also column by column, looking at users. In a full matrix, the SVD algorithm develops a much closer prediction ability, because the values allow it to approximate whatever function it is that the matrix would embody. So, one movie that has high ratings for many users could at some point drop significantly for some other "type" of users. The problem is that there's not that many measuring points to find that out.

The harder predictions are those that fall far from the average that the algorithm generally would predict. But since those perceived values are rare, the features do not really embody such strong divergence.

I actually found that the key to solving everything without the use of blending is to do more work up-front, to better approximate the actual function, such that SVD can fill in the rest. The earlier you descend down to the real values (the rmse decreases), the better it is overall.

I've also seen posts from other people where they seem to be able to leave the SVD algorithms run for 1,000 features. In my implementation, I clearly see that there's a point where it develops divergence, so the rmse increases. In my original implementation that was around 32 features, right now it's about 42 features. I've decreased learning rate and improved the parameters and surely I'll post those once I see significant improvements.

Overall, if you're not significantly changing the heart of your algorithm and do not use ridiculous parameters, this point where divergence occurs is more or less the same. But also the improvements in rmse from one feature to the other remains more or less equal. Thus, if you are seeing a pattern of 0.176155, 0.007401, 0.004668 as improvements, if your rmse is lower for another round because you improved parameters or did more work, I've found that the descent is more or less the same, thus reaching an rmse that is as low as the difference in improvement of the first feature.

Using wilder parameters (much lower learning rate or stronger regularization) can make things look really good from the start, but divergence or instability is much more likely to occur after 15-16 features or so. Thus, I'm not tweaking those parameters that much anymore, as the strength of SVD isn't so much in the parameters, it's in how things are learned and what you do before/after. The differences for good values of lrate and k are only relatively marginal.

The learning rate only affects how fast the algorithm learns. But slightly lower rates allow your rmse to descend deeper, because it realizes a better approximation of the function. I have found though that changing K throughout the svd run has some positive results. Apparently, data easily overfits and increasing the value of K gave me some good results up to feature 14, yielding instability afterwards, but that was probably due to hitting the boundaries of its effectiveness.

I've calculated that, if performance really is linear with regards to the starting rmse, I'll need to start with an rmse of around 0.8957 in order to hit the 10% mark. So, by just looking at the first feature, and assuming that this performance is linear, I can easily see the results of my changes to the algorithm.

(EDIT: See results. Still rmse was lowering, so will reproduce another run with some other small tweaks and a larger change). More tomorrow!

Chasing the Netflix prize

As I'm naturally interested in algorithms, especially when these are related to Machine Learning or Artificial Intelligence... I'm posting some results here from my research into the Netflix prize so far. My results so far are not very impressive, but I'm gaining lots of new insights into the behaviour of Machine Learning algorithms and the relations between parameters, what they do. You can see on the left an attempt at lowering the Root Mean Square Error, or RMSE. That is the key feature to reduce in the Netflix prize. RMSE is calculated as follows:

1. Predict the number that a user would rate a certain movie with one decimal accuracy.
2. Compare the predicted number with the real rating that we know (training data).
3. Multiply the error with itself (square) and add that to a sum.
4. Divide the sum of the squared errors by the number of predictions made.
5. Take the square root of that sum.

Netflix initially had a performance of 0.9514 rmse. Their challenge is to improve this by 10%, thus 0.8563 rmse.

In the picture above, you can see a blue line, which is a very potent reduction of rmse on the training set. It's actually a perfect logarithmic regression, and it follows that line perfectly. However (and I don't have sufficient probe points in red in the graph to demonstrate this), the performance on a set of data that is unknown (a probe to test against or the qualifying set) is gradually decreasing. Meaning, for data that is unknown to the algorithm (the predictions to be made) are getting worse, much worse. That means that the rmse for the probe and qualifying set is getting larger as the algorithm progresses. Thus, the training set is overfitting the parameters very heavily, yielding negative returns for the other data sets.

Looking at the projection of the red line, the rmse of the separated probe (known ratings, but excluded from the rating set) is not converging at some point. My conclusion is that I need to think of something else.

In some of my later iterations that I'm not showing here, it's interesting to see how the performance of the dataset is logarithmic, whilst the performance of the rmse on the probe/qualifying set is neither linear nor logarithmic. The start of the probe rmse is sort of linear, but possibly will become logarithmic after some time. At the moment, I'm at position 1120 on the leaderboard, but I've only touched the surface of this problem and I submitted those results based on a Matrix Factorization algorithm (MF) with only 16 features. As a comparison, some other people have submitted results using over 600 features, calculated over 120 epochs (runs for one feature) or so.

Understanding the value of Data Mining

Artificial Intelligence is quite a large area, academically speaking. Truly accessible applications for business are not so common however. Data mining may be one of them, but as a manager you should approach data mining correctly. Some people may tell you that having a data warehouse is the first pre-requisite to be able to undertake the first steps in data mining. Other managers will tell you they've gotten "SQL in their fingers" and will be able to come up with some interesting discoveries given some experimentation time. Some lucky miners have access to a large data warehouse and just run a few queries to confirm suspicions or new theories.

The thing is, data mining is not like a hobby, it's a true profession. You'll know the difference between a hobbyist and a professional when you ask them how they think about data mining. Find out if they think it's about developing a hypothesis and then testing the data against it, or whether it's discovering new truths about data. True data mining is guided by business perspective. Let someone from the business tell you where it aches or where they want to improve and then go off to find their answers. Are customer leaving? Does the business want to become more efficient in one area? What is the knowledge they lack, rather than the knowledge they want to confirm?

Data mining, to be efficient, needs a focus. It's easy to tell a company that the only way to even consider commencing in mining is having a data warehouse. Time has proven that the establishment of such a DWH is time-consuming and doesn't provide the necessary payback because half is not used.

Factor in your business goals with these requirements. What do you want to achieve? Let an expert go over these goals and consider the data flows that are needed. Then just get those data flows from wherever you can get them, load them in a database (general ETL), don't necessarily model it out, especially if you need it only once, derive your conclusions and move on if it's not giving the payback you require.

Especially in these times, managers need to be more focused on the evaluation of what provides payback and what doesn't. Don't linger around, find out which things are promising, eliminate those things that cost money.

Data mining can be very strategic also for not so established companies. Once you gather sufficient volumes of data, you can start considering it, but you definitely need expert guidance here, it's not a job that the common software engineer can get away with.

Worst of all, don't rely on anyone offering "neural networks" to tell you about things. Make sure to use proper algorithms to smooth and massage your data, so it becomes more interpretable for human beings. Graph it out, since that's the best way to visualize complex data sets. One picture is better than 5,000 numbers.

And finally, it's very, very unlikely that any data mining algorithm will tell you: "Please do A in order to achieve B". In general, data mining results require interpretation and understanding, especially understanding about its limitations.

Developing your custom kernel with qemu

I'm preparing a presentation that is hopefully going to be shown at an internal event at Sogeti. We're discussing the internals of Operating Systems. For this presentation, the intention is to compile a pre-created, very simple and basic kernel, which doesn't do much but print "Hello World!".

Well, if you're interested in doing kernel development yourself, there are plenty of resources around to help you get started. The not-so-easy thing is getting your development environment in order, so that you can actually run and debug the kernel. Here's a set of commands that help you do that. I'm using qemu as a VM emulator, gcc and nasm for development and Ubuntu as a host development system.

1. qemu-img create myos.img 32M
2. losetup /dev/loop1 ./myos.img
3. fdisk -u -C65 -S63 -H16 /dev/loop1
4. ( create a primary bootable partition across the entire disk)
5. losetup -d /dev/loop1
6. losetup -o32256 /dev/loop1 ./myos.img
7. mkfs -t ext3 /dev/loop1
8. mount -t ext3 /dev/loop1 /mnt/image
9. df -Th
10. losetup /dev/loop2 ./myos.img
11. ln -s /dev/loop2 /dev/loop
12. cd /mnt/image
13. mkdir boot
14. mkdir boot/grub
15. cd boot/grub
16. cp /boot/grub/stage1 ./
17. cp /boot/grub/stage2 ./
18. cp /boot/grub/e2fs_stage1_5 ./
19. grub
20. grub>device (hd0) /dev/loop
21. grub>root (hd0,0)
22. grub>setup (hd0)
23. grub>quit
24. vi /mnt/image/boot/grub/menu.lst
25. menu.lst:
    timeout 0
    default 0
    title minimal­kernel
    root (hd0,0)
    kernel (hd0,0)/minimal­kernel
26. rm /dev/loop
27. losetup -d /dev/loop0
28. losetup -d /dev/loop1
29. mbchk kernel.bin
30. cp kernel.bin /mnt/image/minimal-kernel
31. umount /mnt/image

And then run it with:

qemu -m 100 -hda myos.img -boot c -no-kqemu

There you go! Of course, once the image is created and sorted, for development you only need to repeat step 30. You can remount the image in a build script:

mount -o loop,offset=32256 ./myos.img /mnt/image

Here are more resources:

• http://www.osdever.net
• http://www.jamesmolloy.co.uk/tutorial_html/
• http://www.osdever.net/bkerndev/index.php (Bran's kernel development tutorial)
• Intel Architecture Software Developer’s Manual (search Google)
  
• http://oreilly.com/catalog/linuxdrive3/book/index.csp (Linux device drivers, 3rd edition. FREE as in beer!)
• http://tldp.org/LDP/tlk/tlk.html
• http://www.ibm.com/developerworks/linux/library/l-linuxboot
  

Physical Symbol Systems

At the university, I'm now following a course on knowledge systems. There's plenty of room for one to be sceptical about knowledge systems, in the sense that it's possible to think of general procedural programs as knowledge systems too, the knowledge system is identified by the ability to separate knowledge from processing (the inference engine). This great distinction, if followed to the letter, means that knowledge systems are only valid once knowledge is explicitly declared and not mingled with source code of the inference engine. This rules out most procedural program implementations, if not all. In a simple interpretation, I'd say that knowledge systems can only be built in Prolog.

I don't want to discuss this item for the rest of my post however. What is more interesting is the hypothesis stated by Alan Newell and Herbert Simon (picture above). They stated:

A physical symbol system has the necessary and sufficient means for general intelligent action.

This implies a couple of things. First, that intelligence can be thought of as symbol manipulation. Second, since computers can be thought of as symbol manipulation machines, computers can in theory become intelligent.

Many efforts on representing knowledge in computers generally start with some written-down term of a particular symbol. Reasoning with text however is a bit daunting, that is, the symbol itself cannot be broken down and its meaning is invoked through its connections with other symbols and concepts.

Thinking very abstractly however, symbols are representations of knowledge, and symbols need not be visual. I could for example describe a cow as 0x7A832B12 and add further different representations of the same thing. Colloquially, I could add the image of a cow, the sound and the smell and tie them together as representations of the same thing.

Symbol reasoning systems then require the computer to categorise the symbols themselves, to find ways they are equal, ways they are different and how symbols might be related to other symbols and in which way. It's even possible that the relation itself is yet another symbol.

A limitation of our thoughts might be that we rely on our language too much in order to be able to debug knowledge systems. I'm considering that language itself might not be the most efficient way to develop reasoning systems.

Another important quote that I read today is that there is no true way yet to assign meaning to symbols, however meaning can be represented in a computer, that is.

Hmm.... the ideas I have about this subject are really abstract and it's almost impossible to write them down in a sensible way, at this time. But why rely on our or any other language for a representation of our knowledge? If knowledge has different appearances, then shouldn't we let a computer choose how it decides to store it? The designer of any program makes full decisions about data, and data structures.

For learning machines though, we may need to innovate on how data and knowledge is stored, such that more complex systems could possibly use it in different ways, hopefully with the ability to derive new knowledge from existing knowledge, which seems a current hard limitation for a computer at this time.

So... goal: Build a knowledge system without a specific design for the storage of knowledge, or built it as a hybrid combination of implicit knowledge with explicit knowledge.

The world explicitly in "math vision"

Computers don't have sense of anything. They're basically only good at processing according to a couple of defined guidelines (the program) and nothing much else really. It's just a processor, like a blender processes food and was designed to do so.

Artificial Intelligence is so interesting, because it comes up with novel ways to reason with input. In a very narrow (or is it broad?) definition, any computer program is artificially intelligent, because any program uses "if-then" rules. However, engineers typically do not accept that definition, because the intelligence conveyed by such programs are not surprising and do not supersede our own capacity for reasoning. Popularly speaking, being intelligent means that someone, some animal or something is behaving in a way that surprises one.

A (regular) program cannot execute other rules than those "if-then" rules it has been designed to handle. That generally makes it very explicit and consistent in its behaviour (unless bugs are in the program). Let's assume that any program I'm describing here has been 100% tested and is guaranteed 100% bug-free.

The world to physicists and mathematicians looks different in many ways. There's a constant awareness of approximations of behaviours through formulas and the awareness that some problems that look ridiculously simple are astoundingly hard to solve or describe mathematically.

For A.I. to continue into its own field, it'll be an ongoing battle to get the computer to reason and "understand"? (this latter term should be used very carefully) its environment better. Roger Penrose highlighted four different viewpoints on the mind (mind != brain), where on one extreme the mind is 100% mystical and unexplainable and at the other extreme it's 100% computational. Dr. Penrose is a physicist and he doesn't seem to be inclined to believe that the mind is 100% computational (viewpoint A), but there's a strange missing link that allows computational processing machines to become aware (although one could also argue that awareness actually means introspection abilities), the state vector reduction between quantum physics and that of classical physics.

When one mentions "make the computer smarter", one generally assumes that the computer should become more like us.... but thinking about it... there's no reason why it must or should. I've argued before that humanity is pretty arrogant when it comes to words about intelligence and it basks in the light of its own narcissistic tendencies. Consciousness is not truly a pre-requisite for life-like intelligent action. Although consciousness itself is very likely not achievable in non-biological machines... can some sort of consciousness be simulated or modeled?

Some posts back I wrote about rule mining. For a computer to simulate consciousness, it should be able to deduce new rules, descriptions by analysing its perceptions. However, in order to even start doing that, it must see the importance of doing so in the first place. And in order to see the importance, it must understand the context and environment. So this is a circular argument, seemingly? Well, we certainly don't get born with objectives and understanding from day one. So there's a learning element involved and impulses that determine our goals. Baby's do have certain goals, although simple: "eat, poop, sleep" and they'll keep on crying until the goals are satisfied. Babies are human, but popularly, we're not considering them conscious yet. Possibly we start considering little children conscious when they start talking...?

Here's an account of a professor who has autism. The read is very interesting, especially the last summation, where four different levels of consciousness are given:

1. Consciousness within one sense.
2. Consciousness where all the sensory systems are integrated.
3. Consciousness where all the sensory systems are integrated with emotions.
4. Consciousness where sensory systems and emotions are integrated and thoughts are in symbolic language.

Next to the brain, we're also responding to chemical changes in the brain. In fact, just as the ear receives auditory information and the eyes receive visual information, we could think of chemicals and proteins to produce chemical information for our brains to process. Those proteins and chemicals indicate our own state to ourselves, besides the faster processes like pain (if pain were transmitted through chemicals, it'd probably take at least 20 seconds before a response occurred, not efficient!).

Quite some time ago, I mentioned that emotions are the driving forces behind humanity. What I really meant was that without emotions or feelings, we won't feel any urge to start doing anything. It's like a computer on a desk with 0% CPU usage, 0% disk I/O. Only when the proper impulse is given, is the goal generated and will we start to find ways to achieve those goals.

So it naturally follows that a 'conscious' computer should have goal-generating abilities to function more like an animal. The problem here is that one doesn't just code a "goal-generating algorithm". Different people pursue different goals. It depends on experience, outside stimuli, upbringing, different chemical compositions, talent, preference... So it's something that more or less 'grows on you'. How can the same thing be grown in an A.I.?

At a lower level... the building blocks that make US tick and develop the preference in the first place... what is it? I mean, what actually did develop consciousness or shaped it into being? If we assume babies are not conscious, then something at a lower level is developing/placing something there.

This suggests a lower level of being that is using consciousness as a tool for achieving its ultimate goal(s). That goal may be very simple (survival? pro-creation? endless aim for better well-being?), but through our conscious "processing"? layer, it translates this into several different sub-goals.

Presentation Sogeti Engineering World 2009

On Saturday, the 7th February, I'll be doing a presentation on Artificial Intelligence at Sogeti Engineering World 2009. I'll be talking about the Singular Value Decomposition algorithm, ID3 decision trees, the Bayes theorem and I'll dispel a couple of myths about neural networks. You'll need to register to go there (aanmelden).

Microtubules and consciousness?

I've finished reading "Shadows of the Mind" by Roger Penrose. It was a very interesting book with interesting views (the view from physics). Dr. Penrose enters into a long explanation with arguments on his theories, one of the most interesting being the role of microtubules in consciousness. At the end of the book, Dr. Penrose asserts that consciousness cannot be invoked by machines, devices or biological entities which are only composed of computational algorithms and actions. That is, he asserts that something non-computational needs to become part of the equation in order for consciousness to exist.

The arguments are compelling in the book. If consciousness is evoked not by the neurons, but by the smaller microtubules that are part of neuronal cells, then the computational power of the mind exceeds the computational power of computers even further, by a factor of 100,000 or so.

Although at the same time, I'm not so sure about how this theory holds. The book is very explanatory about quantum theory and mechanics and explains a number of different puzzles and examples in quantum theory. One of the key questions it poses is the state vector reduction problem (collapse of the wave function), which is the process inbetween the quantum world and the classical world as we experience it.

Another thing I did not yet see anywhere is the concept as in the previous post, the likelihood of algorithms that influence one another. Thus, rather than a single algorithm which is executed by a single thread or CPU, is consciousness actually the collection of calculations in different threads / CPU's at the same time?

The very interesting thing of the book, if the consciousness is evoked by microtubules, is that neurons are then clusters of calculations, which influence other clusters. Like macro-signals of tiny little calculations that are then sent to other positions where the information is used as input for further calculations. It also may have some relevance to memory?

In connection with other posts, I have written about consciousness and reasoning as be it some kind of fluid algorithm, where the possibilities and concepts are tied loosely together as some kind of oil, with the thread of thought passing through it guiding the selected items. Items that are connected to others on the thread may appear in thoughts, given certain changes in context.

Then we could also make the point that, if Penrose's ideas are true, that microtubules are able to evoke any thought whatsoever, where the choice for the exact thought or idea to come up is made through some sort of calculation or determination. Thus, just as in quantum theory, the thought is not clear and could be any, but through a range of filters or possibilities, the final thought is evoked by the final filter.

The Matrix Aggregate

Matrices are mathematical tables, which are used to record elements of data in the world around us. These are widely used for example in keeping track of rotation and translation operations like "SLERP" in 3D computations for games or simulations. Matrices are also used in the Singular Value Decomposition and have many other uses. After the recording of data in (possibly huge) matrices, one can perform various operations on the data, often resulting in a destination matrix that conveys a certain meaning.

Matrices thus are very interesting for Artificial Intelligence. It can operate on large datasets with the objective to process that information into something new, which then is used as a shortcut for making predictions for example.

A limitation of matrices is that all the information for a single timepoint or a range of timepoints needs to be available. This is often very difficult to achieve, or the resulting matrix may become so large that the general PC struggles with available memory to perform the computations.

Many academic texts written on consciousness and artificial intelligence are written from the perspective of the computational mind. But they are also written from the perspective of an algorithm. Since most (if not all?) algorithms are serial, this also suggests that the mind or the brain is serial. This is certainly not so, each neuron can fire independently in time and need not be given any CPU time for the neuron to actually fire and influence other neurons.

This suggests a parallel nature as large as the number of neurons available in the human brain. So, not only do we have more neurons in the brain than the common computer can hold by itself (not even counting the memory needed for maintaining connections), each neuron also operates as if it were a CPU by itself.

It's certainly the case that some algorithms can be parallellized, therefore allowing them to run on different devices and then have their results combined to find the answer. This is what is meant with parallel algorithms in the field of computer science.

Here though, we should also consider parallel algorithms to be algorithms that are truly parallel in nature, algorithms which run on many different processors and operate on the same data.

Just recently, I wondered what would happen if some sort of chemical concept were introduced in ANN's. Thus, an ANN would not just execute on neurons, thresholds and biases to find new values, but one could introduce chemicals that would change how neurons fire in the ANN. The applications of this aren't really clear as of yet though.

More next time about this topic.

Wisdom: Rule mining

Artificial Intelligence is very likely to gain a lot more traction in the coming decade. I think it has already started. A.I. is not a science that is solely concerned with rebuilding the human brain or just a couple of cognitive aspects. A.I. is also related to questions that pertain to interaction of agents within a society or organization. In that sense, it's trying to combine individual decisions, individual cognitive abilities with the cognitive abilities and behaviour of that organization as a whole. A.I. is already a multi-disciplinary field of science with strong links to computer science, mathematics, philosophy, psychology, cognitive science, anthropology, management studies and possibly a couple more :). I'd like to start making the claim that A.I. isn't actually domain of computer science. I see it primarily as the science of conversion of problems in other spaces (psychological, behavioural, analytic, business) into expressive models that can run on computers. So it does share a lot of "knowledge" with the computer sciences, but for A.I. it is only the last step. C.S. on the other hand has many topics which are solely related to how to run something on a computer faster or more efficiently, so it is constrained to the elements of C.S. itself.

Thus, in other wordings, A.I. Computer Science as a means to offer a model or simulation of reality. Computers are good tools to use, since they have the capacity to process mountains of information in easy steps.

A big challenge of A.I. is the A. part actually. This A. part deals with computers that only accept an explicit and deterministic language, something that we're not exactly used to. A computer was designed (although it doesn't always behave that way :) to be 100% deterministic. Every cause and effect must be clear. In other words, every cause or event needs to have the intended effects and every observed effect must have a perfectly explainable cause. Challenges here are still abound in cases where events are not received or effects occur that seem inappropriate to the current context (the system is acting weird).

Yet the world doesn't always act in a deterministic way, and we don't use the same deterministic language within the same organization, not even within a single relationship. As soon as someone tries to impose a single perspective about "how the world works" on an organization, it somehow starts to fight back. Slightly different interpretations work better in different contexts. Computers can't deal with that though, since it's not truly contextually sensitive.

In this case, strategies that are based on more fuzzy representations of data sets can work better. The problem in those fuzzy strategies is that a computer can't derive rules from it. So, it doesn't actually gain any knowledge, other than a mathematical representation of how something works.

A very interesting academic exercise though would be an attempt to mine for rules in fuzzy data sets. Suppose that a system finds that customer A likes video A & B & C, and customer B likes video D & C & B, what are the common properties between those video's and how are they important for purchasing decisions? Can we actually profile customers in non-mathematical terms in this way and make statements like : "customer A likes action movies, except not with Keanu Reeves as a main actor?".

The establishment of such rules requires a lot of knowledge about the concepts that a computer is dealing with. As an example, SVD is an algorithm that can be used to analyze preferences or "like-ness" of books or video's. But it cannot state anything in our language about those concepts. If it were possible to construct phrases from such analysis, then also the computer could use that knowledge to develop (executable) rule sets.

Or maybe we shouldn't start with analysis in the first place, but start with rulesets. Develop a hypothesis and test that hypothesis (by how far it is true) through the mathematical analysis?

The ability for a computer to switch between sets of rules and mere "analytical processing", even though it was not programmed to do so in the first place, should be a very important area of research for the future. Learning for human beings is also about assimilating "knowledge statements" from our peers and then testing whether those statements are true by testing it against our experience of reality.

The outcome can be:

• No experience on the topic, so I cannot verify if it is true or false. (insufficient evidence to validate your claim).
  
• Insufficient knowledge in parsing the statement (I don't know what you mean, could you rephrase that please?)
• That sounds interesting. Indeed I have some evidence that suggests your claim is true. Can you give me more examples?

A rule strength rating should probably also be given. We'll often find outcomes that contradict the rules. In those cases, we could be missing "except-if" cases or "and-A-and-B", where we failed to observe B being true most of the time, except for the last case where it was false.

How do you design a rule-based program that isn't as explicit and hard as Prolog for example, but more like a "soft-rule" program where it accepts statements that are generally true, but not necessarily always and where the computer can verify for itself the strength of those claims as well as form others based on observed data?

Quantum mechanics and consciousness

In quantum mechanics, the very act of human observation changes reality. That is, by observing something, you are having an effect on what you are seeing. Sounds strange, doesn't it?

Once, I watched these "ants" at work when carrying a piece of lime up a wall to their hideout. Wondering how on earth an ant society that has limited ability to communicate between themselves (pheromones are the main way of communication), they still apparently have other ways of communication. Why? Because the piece of lime is heavy! Even for an ant. It still managed to work with 20 or so ants together, carrying it up hill. But how does one ant know that another is getting tired? Do they need to?

This blog post here is about "entangled minds". It is an explanation about how we use intuition as another means of sensing the world around us. If you think the idea is ridiculous that minds can be entangled, and that at a sub-atomic level, things may interact in a different way than we think imaginable (that is, not like "matter"), then...

How about this... Do you ever get this eery feeling of being watched? and when you turn around or look around you, there *is* someone actually watching you? And you can sense this even when you are with your back turned to the person?

Or how do you explain that people in different parts of the world made historic inventions roughly around the same time?

Going back a couple of posts, I considered the point that consciousness is actually nothing more than an observation of mental processes that will happen no matter what you want or do. Thus, a perspective on the brain as if it were a giant, powerful computer that executes things no matter what, can be influenced by drugs (by inhibiting or stimulating receptors on cells), which on a higher level receives pre-processed input from our senses. It is also influenced by those ideas in quantum mechanics like the observation problem.

If this holds, then just as matter could be slightly more static, us as human beings are actually also part of that matter, the table in front of you and everything around us. If we are not as much in control of our thoughts as we think, then our thoughts are possibly also controlled by those things around us. Thus, as some movies make us believe, by observing something it falls together into a single state. But by not observing it, some object could be anywhere, anyhow and anytime. That is, the possibilities are that it is in all places at the same time. When we turn around and observe it however, it becomes static.

Now... we should probably not assume that it makes us a superhero, but maybe the idea is that by observing it we are interacting with it at a sub-atomic level.

This is surely mind-numbing to think about. We'd like to think that we're discrete, individual personalities that make up our own minds about things and have our specific achievements. But then we find out that we're just part of this big mess in a very different way.

If the above holds, then what does this mean for "causality", "determinism"? What patterns of control are there at this sub-atomic layer? How does that system keep itself in balance? And are electron patterns or executions in a computer ever going to reach this quantum state, in such a way that when a computer observes something, it is interacting with the environment? And if it is not, how is that going to affect the computer's effectiveness? Without intuition, will it ever be able to interact with an environment at all?

If we push this quantum space out of balance, will it push back?

Singular Value Decomposition

An extremely nice tutorial about Singular Value Decomposition shows how you can extract pretty specific information from a bunch of data. I think SVD is very interesting to analyze data from different perspectives, one perspective is the product (how close is it really to another?) and the other perspective is the customer (how close is customer A to customer B?).

The problem starts to occur when people change their preferences. People generally go through phases (well, not all of us, but many!), and this is accompanied by different needs and different preferences. For this reason, A.I. designers need to understand that historical information only has limited value. The temporal trends in such analysis never come up to the surface, but I'm sure that some research is being done in this area, to further contextualize data in the realms of time.

I'm likely to speak at Sogeti Engineering World 2009, yet to be confirmed. My presentation will be about Artificial Intelligence and how it applies to business. Already now, businesses at lower levels are getting more interested in making the most out of their data. They have good knowledge about how their business works and who (in general) their customers are, but they cannot quantify their customer base from different perspectives.

My presentation will make clear how Artificial Intelligence is important to cases like response modeling, online recommendations, retention modeling and it will explain to engineers how they can apply certain techniques (borrowed from libraries) to their own problems at hand.

Where most people think of A.I. as some kind of black magic or silver bullet, I think it's important to realize that it's just juggling with numbers (at the moment). Over the past 50 years, A.I. has expanded into a number of different territories. One territory is more related to our "explicit knowledge" about things, the rule based systems and prolog. The other area is more related to "tacit knowledge", or what we know without being able to tell how we know it. It just works/is.

Neural networks, SVD, Kohonen are more mathematical constructs around the idea of tacit knowledge. We can't really trace it from input -> output, we just know it works. Other languages like Prolog work on the execution of basic rules or truths and demonstrate how the real world would act.

Our minds continuously sway between these two different areas of knowledge. We infer a lot of different information just through observation, sometimes supported by external teachers. But we also judge observations on truths that we have learned, or rules.

Many solutions in A.I. have depended on the combination of different techniques to offer the best solution. One solution that seems to work well now, for example, is spam assassination. SpamAssassin, now an Apache project, is one of the most popular spam-fighting schemes for email servers. It doesn't depend on a single scheme to rule out spam, but combines them as part of a certain model. Each different technique is either restraining or backing up another technique.

The very interesting question here is that in computers, we tend to use either RBS (Rule Based Systems) or other techniques like Neural Networks or Bayesian Belief Networks to solve a certain problem. One system is invoked before the other, as in a type of hierarchy. If we assume that the human brain only has neurons at his disposal, how can all these different techniques be applied in unison at the right time and moment? How do we know which strategy to rely on?

linux intrepid tricks

I've upgraded to Intrepid recently and just two days ago, my system collapsed. For some reason, while opening a new tab in Firefox, the entire system just stopped functioning. No terminal, no Shift+F1, no login... So I reset, expecting things to resolve itself. Naturally, the reboot entered "fsck", which found a number of errors. However, I couldn't leave the machine working on that since I had to leave. In the evening, I tried things again, but it got slightly worse. It took 1 hour for a single fsck run with loads of messages inbetween. By then, I was thinking that I could reduce the time for fsck by removing a DVD dump from one of the DVD's I am owning. Bad idea. As soon as I restarted and went into rw mode, I got grub "17" errors on restart. That means that the boot loader can't even resolve the partition to boot from.

I did have a live cd lying around somewhere, but that was not of great help. "cfdisk" absolutely refused to run. I could not mount from a terminal in the liveCD

( mount -t ext3 /dev/mapper/isw_xxxxxxx /mnt/target )

resulting in "superblock errors" or "partition could not be recognized" and those sorts of things.

and from within "grub", it couldn't even see /boot/grub/stage1. setup (hd0) didn´t work either.

Well, searching around on the internet seems to regularly suggest to use the "grub" trick, or suggests that the root (hdx,y) setting is incorrect, but my problem clearly was a hosing of the entire file system. I thought.

Well, since I am running from a fake RAID array, I needed to remember to install "dmraid" (intrepid has this by default now), but in feisty that needed to be activated through the sources.list first, then apt-get updated and then installed. Then perform "dmraid -ay" to get the /dev/mapper devices to work.

It makes no sense to mount a RAID-ed partition directly through /dev/sda2. You should remember that as well :). On the internet, I couldn't see very good pointers, but eventually I decided to finish where the single user mode left off: fsck.

root@recife# fsck -y /dev/mapper/isw_xxxxxxxx02

eventually ran the entire file system check and resolved looooads of errors. Mounting this on /mnt/target later did work. I could also sort of boot into the system, but because /etc was gone, it wasn't very helpful :). So the entire system got hosed, but from the Live CD system, I could rescue a couple of important files and put them onto different systems or mail them around. Thus, I didn't lose my university assignments and what have you, but the entire installed system is a loss.

I've now re-installed intrepid from the netboot cd (download from the internet) and that worked in one go. There's a guide on the internet on how to install that for fakeraid systems. It's a lot easier. Grub however still has problems getting things organized, so you should pay heed there. Also, it seems that "update-grub" doesn't work properly when menu.lst does not exist. Actually, it does attempt to ask you if it should be generated, but that doesn't work well. I ended up creating a single file "line" with a "y" in there and then adjusting the /usr/sbin/update-grub script (line 1085).

On reboot, things already worked fine, but I like to install the nvidia restricted module drivers for better performance. The screen resolution for my IIyama was still problematic though. It only got to 1024x768. Eventually, I ran nvidia-xconfig, which put in more crap into xorg.conf, then restarted xorg ( nohup /etc/init.d/gdm restart ), after which I had more options to choose from.

Right now, I think I've more or less entirely upgraded to the system I had, so I can carry on hacking and doing things. For some reason, the old system was slowing down significantly. And then there's not even a heavy registry to be supported.

Mental Causation

An old philosophical problem is the problem related to mental causation. The question relates to how a mental event can cause physical events or whether mental events are the results of physical events. In my previous blogs, I once posted about how clever we think we are. This post is sort of an extension on that. In the post, I pointed out that we consciously often consider ourselves more intelligent and better than other species, but our actions are not necessarily that much better in regard to action -> consequence. It's just more words and more fluff. In short, we easily believe that we're radically analyzing a certain situation, considering it from any angle, objectively, but when one uses hindsight to analyze the situational developments later on, we often see that the original arguments were severely misguided or didn't have any such intended effect.

In my studies, I'm now following courses on modelling. The A.I. classes are divided into a group following Collective Web Intelligence and another is following Human Ambience. The latter requires to understand more stuff about decision-making, well-being, psychology, sociology, altruism and so on. You wouldn't possibly exactly expect it from courses in A.I., but there you go.

It's intensely interesting. One of the courses today is about emergence, which I also blogged about before. Emergence is about simple constructs which act/interact in rather simple ways, which eventually construct a new model of behaviour at a higher level. Ants are the most common examples, where each individual ant follows a couple of simple rules, but the behaviour of the ant-hill overall is far more complex than the sum of individual ant together.

You could consider the mind not having any actual conscious thought at all. A not-so inspiring idea is to think of ourselves as soul-less beings, within which just run a very high number of different physiological processes (100 billion neurons), shooting off electrical messages between one another whilst being impacted by a couple of hundreds of different proteins, which are messages from one organ to another. So, we have no specific 'soul', we're just like robots with very complex physiological processes, eventually yielding a certain behaviour that allows us to interact with others.

The ability of a neuron to form an electrical current than is the physiological level. Let's call this emergence level A. But by forming this current together with a simple method for recognizing a previous pattern (neuron A firing off and then neuron B responding similarly because it has done so before, also known as strengthening of a synapse), is a cognitive process, where it doesn't just become a process of firing electrical currents between neurons, but a more complicated process of responding to certain firing patterns. Let's call this emergence level B.

(We then need to take a couple of too quick steps by jumping to enormous assumptions and conclusions) If we assume that thoughts are somehow emerging from these patterns of firing neurons, then the 'memory' together with some other 'machinery' for computing and predicting the results of actions could be seen as the basis of our behaviour. Thus, behaviour in this definition is the ability to recognize and remember and predict future outcomes and then acting on those computations. The next level is our decision-making and behaviour, level C.

When you go one more level up, you get to the behaviour level of a complete society. Remember the ants? For humans, you can develop similar models, because we have a model for our economy (where each of us acts as agents) and a model for certain criminological events, etc. The behaviour of society is made up out of individual decisions at level C, but overall might develop a new emergence level D, that of the collective.

The interesting part in this consideration is that mental processes aren't so much "spirited". From the Stanford Encyclopedia of Philosophy:

(1)	The human body is a material thing.
(2)	The human mind is a spiritual thing.
(3)	Mind and body interact.
(4)	Spirit and matter do not interact.

The above four rules regard the mind as a very special kind of element, sort of like a merger of the soul with some physical abilities that the brain can do (vision, smell, motor control, etc.), but decision making, emotion, etc. are considered somewhat deitous.

If we simply regard the mind as a number of computations that are biologically there and thoughts and consciousness are the de-materialization(?) of certain cell assemblies becoming activated or not, then we can find ways to merge this blog story with certain theories about how DNA is actually indirectly programming us and how we serve as carrying "agents" for the continuation of the DNA structure. Thus, in that sense, we are walking biological computers, which are continuously responding to our environment, learning from it and through those processes become more efficient in the propagation of cultures of DNA.

One can wonder whether our consciousness is really that 'evolved' in the sense that it is the motor of all our cognitive processes, decisions and what have you. Are we guiding our actions and thoughts processes through our conscious 'participation' in this process or is consciousness the reflection of the human brain itself, which has basically already determined the best course of action and has considered each alternative? Thus, in this latter idea, consciousness is more like an observation of "mental processes" that have already taken place or are going to take place thereafter. Thus, the difference here is that we must properly identify the CPU, memory and machine and not point at the monitor screen to describe "the computer". In this analogy, consciousness is the reflection of what goes on in a computer (thus, the image on the computer monitor), but it should not be mistaken for the computer itself, which is generally more out of view, housing the CPU and memory.

What is not explained though in this entire story is the element of attention and how we are able to 'consciously' execute certain actions or pay attention to important things. Is that just a matter of directing more attention and execution power to physical events? If it is, then who's instructing our machine that it is important and should be paid attention to? Is the brain in this sense self-preserving and intelligent that it controls itself? Or is there an externality involved which directs the attention of the machine? Or are we thinking too much in hierarchical terms and is the entire problem of decision-making the problem of weighing off cost/benefit and dealing with direct influences first vs. more indirect influences?