Confabulation and how it relates to SVD?

I've been reading up on this confabulation theory. It's pretty interesting work, the theory, but the practical implementation isn't as nice as the ideas behind the theory :). The use of matrices in the implementation seems to basically make this a problem of statistical analysis. The difference with Bayesian theories and other probability frameworks is that this theory assumes a "causal" relationship between one and the other, not in the way that Bayes does, but through cogency maximalization.

In the end though, and reading the word "matrices" somewhere inbetween, it all sounds a bit like singular value decomposition with a twist, rather than something entirely new. I've been looking for ways to replicate their results somehow. I came up with the following:

• Using Hadoop as an underlying platform for simple tasks (like counting words, ordering them, etc.), you take out a lot of complexity out of a program. So I'm using Hadoop to count word frequencies and the frequencies that words are seen together and at which distance.
• Hadoop provides a platform where I'm mostly using files, intermediate results stored in files and reducers to bring back some sanity. If I'd had to program that myself, I would be more concerned about stability and reliability than the core of this little research, statistical analysis (in short).
• The results are used to train a couple of SVD's á la Netflix - gradient descent. Because I've got a good 1GB file of frequencies, I don't need to process text anymore, it's all ready for learning. (text processing required 5hrs to process and get the results together. The problem in learning is that the statistical significance is only apparent after the entire lot was processed, or you'd have to heuristically make estimates).
  
• The SVD's are about 60M in size. On my machine with memory left, I could theoretically get to 33 SVD's, working together.
• The pre-processed files allow me to process the entire "frequency" file line by line, one after the other. I post-processed it to take out any words I did not recognize.
• Since I know the statistical significance of each word, how it relates to another and by what distance, I can just keep on training the features of each SVD. Not knowing that beforehand makes training slightly difficult (doh!).
  

I also thought about possible results of confabulation if it were implemented as a chat bot. Suppose that the technology were ready to produce sentences as if a chat session was going on. The Turing test is one important test, where judges need to make clear distinctions between humans and computers. Since this is a subjective test, I can't help but think that there's also an element of expectation involved. The judges know what a computer does, how it operates and what its limitations are. Now, if someone invents something really cool that breaks that barrier, the judges may temporarily be baffled and come to incorrect conclusions. But as soon as they learn about the research, they may adjust their expectations, raising the bar for the future.

In short, *can* the Turing test ever succeed? Because what if a computer OVER-performs, giving away its true nature? It seems that for a Turing test to succeed, the computer must be on the exact cognitive level of a human being, not higher or lower.

Anyway... Hadoop is really cool and useful. Having another computer in the network would half processing times. Hadoop + custom programs + post-processing scripts sound like really useful ways to pre-process huge amounts of text before you let a machine learn days on end. Since the end product are files that are in a more "edible" format, it should be a lot faster to do the research cycle.

Confabulation theory

Confabulation theory is a theory from Robert Hecht-Nielsen about the cognitive function of the brain, or in other words the working of thought. It's a theory, not an explanation. Confabulation theory basically works by processing lots of information and then from this information, find out which symbols belong together. Those symbols that are often seen together (and in cases by which distance) is information contained within the network. Confabulation can now produce new sentences by continuously generating possible phrases based on the context that is seen prior to that.



In effect, the confabulation theory uses an architecture that can produce entirely new sentences, which are plausible in the context. The interesting thing is that these sentences are also grammatically and syntactically correct. Thus, the rules of a language seem to be embedded within this network.

This does not mean that the machine 'understands' the language, or that it is conscious of the sentences it produces. I think the results should be considered the production of a thought-less brain, which only has the capacity to produce sentences without understanding what they mean. It's probably comparable to the chinese room producer, where it looks at streams of chinese symbols. At some point, the machine understands the order in which the symbols may appear and which symbols are often seen together. When asked to produce a sentence on its own, it uses this knowledge to produce a sentence.

What interested me in the theory is how this network differs from other networks that can fantasize, like the RBM. The RBM is a network that stores knowledge by looking at things and can then complete the signal it is receiving. This confabulation network is slightly different, in the sense that it can project continuations (say, a hypothesis) that are very plausible. So, if you were building a network that can produce responses to sentences, the confabulation network is likely to perform better. But if you ask a confabulation network to recognize a face, it might have good difficulty and the RBM might be better.

The RBM is a flatter network (judging the entire system) in comparison to the confabulation network. The confabulation network just competes between symbols and modules and always takes the highest value of all, but since the signals proceed from those results towards other networks, it's in a sense hierarchical.

It'd be really interesting to be able to identify specific properties of each network and then see if they can be used together. It's also possible that we're thinking about this the wrong way. The continuous processing of the confabulation network is quite different from others. We like to think from static situation to the next, perhaps the entire thing is more dynamic than that and we should focus on generating states, looping back and reprocessing the results, thus continuously adding more results to some hypothesis.

Since A.I. is also a lot about search in search spaces (think chess!), a neural network could be used to generate a hypothesis step-by-step, until it is deemed that a particular branch isn't going to produce a good result, so that it can be terminated.