Bigsley the Oaf

Against Neural Nets as a Computational Model of Intelligence

Posted in Uncategorized by bigsleytheoaf on June 25, 2011

I just sent this email to someone and decided that it pretty nicely summed up my ideas on the subject of neural nets (with lots of ideas which apply equally well to other modern computational models of intelligence – e.g. SVM’s, bayesian nets, etc.)

I think what we ended up converging on, topically, was a comparison between the evolution of our (society’s) ideas about mind to the evolution of our ideas about other things (e.g. physical reality).

The reason for this was that I was asserting that, by various definitions of “bad” which I was unable to effectively elucidate, neural nets are a “bad” model of the human brain/the human mind. I think that you were trying to justify the progress we’ve made (which has culminated in neural nets?) by referring to the overall progress of science.

Let’s look at the progress of physics which goes (very loosely): folk physics -> “elemental physics” (earth, wind, fire, etc.) -> newtonian mechnics -> relativity -> quantum physics -> string theory.

In my way of interpreting science, what has really happened here is a long process of growth and refinement of a finite ontology. We start out with a very coarse ontology:

Folk physics: (stuff)

It grows:

Elemental physics: (earth, wind, fire, water)


Newtonian mechanics: (matter (earth + water + wind), mass, force, momentum, energy (fire…kind of), E&M)


Pre-relativity: (aether + newtonian mechanics)


Relativity: (energy + mass are combined, so are momentum + energy)


Quantum physics: (sub-atomic forces, particles + waves are combined, birth of particle physics (sub-ontology of sub-atomic particles))


String theory: (well… I actually don’t know that much about string theory)

Anyway, the history of physics has been this really complicated and boring struggle to find just the right number and combination of concepts to compactly describe what we see around us. The two operations – adding an element to the ontology, and combining two elements of the ontology – were used over and over in a complicated way until we arrived at our modern understanding.

I’m not asserting that this is the correct way to understand physics – I’m just saying that it’s my way of interpreting what I know of the history of physics. Of course it’s much more complicated than this very linear picture, but this is more or less the backbone (in the sense that it’s what is taught as the history of physics, and each of these theories is still easily accessible, informationally – unlike theories surrounding flogistam, the aether, etc.)

Now what’s key here is that I would assert that each successive ontology is “better” than the one before it. How is it better? I’ll try to describe the intuition:

The addition of an element to an ontology allows it to “breathe.” Once we add “sub-atomic particle” as something we can think about then we have this interesting combinatorial calculus which justifies the behaviors of objects which heretofore were seen as “atomic” (in the sense of being indistinguishable and composed of exactly one object).

The combination of two elements in an ontology makes its representation more compact – and easier to calculate with. Why think of energy and mass as ultimately being separate things, when they can be thought of as the same thing? Sure, in some situations you can, for convenience’s sake, treat mass purely as mass (in the newtonian sense) and energy purely as energy (in the maxwellian sense), but there are situations (e.g. when things are going very fast, or when things are blowing up) in which it doesn’t really make sense to think of these as different things.

Conversely, the unnecessary addition of an element or combination of two elements is quite burdensome. Imagine if, for instance, someone claimed that energy IS gravity – it would be so hard to wrap our minds around this. Or if someone claimed that there was a fifth force, and tried to justify it.

To sum up: do not try to solve an N-dimensional problem using an N-1 or N+1 -element ontology.


Bringing this back to terra firma:

One issue with neural nets is that I believe that the mind is an N-dimensional object and that neural nets present (in my opinion) an N-m (for some m > 0) -element ontology in attempting to model it. Why?

Well, I’m assuming that the theory of neural nets is really just a 1-element ontology (where the element is the “neuron” or node). Correct me if I’m wrong (or interpreting this strangely).

For one thing, on the cellular level, there are other computational structures in the human brain – namely glial cells. Astrocytes (a common type of glial cell) form a syncytium (effectively a large a multi-nuclei cell), release neuro-transmitters, are electrically active, and are about as plentiful in the brain than neurons (there are about 10^11 of each). Here is a pertinent quote from wikipedia:

“For over a century, it was believed that they did not play any role in neurotransmission. That idea is now discredited; they do modulate neurotransmission, although the mechanisms are not yet well understood.”

This suggests to me that we might want to consider a model with (at least one) other element -> it seems to me that, to represent the cellular structure of the brain, we might need at least 2 (if not more) elements in our ontology.

Another issue with neural nets is that they are trying to model the mind in terms of cellular structures. I don’t believe in spirits or god or magic, but this approach seems to me like trying to figure out chemistry in terms of purely quantum-theoretic or string-theoretic mechanisms.

Namely, macroscopic models exist and are useful.

The ontologies of chemistry have evolved along different (though not entirely disconnected) lines from those of physics. E.g. they started with atom -> plum pudding model -> bohr atom & elements -> quantum atom -> etc. etc.

Similarly, I think that what we have witnessed (mostly in the fields of philosophy, psychiatry, psychoanalysis, cultural anthropology, to the extent that they have been performed responsibly and scientifically) historically has been the evolution of an ontology of mind. It is composed of words like “memory,” “computation,” “thought,” “idea,” etc. If the mind has the brain as its sole substrate then these models should be equivalent in the sense the chemistry and physics are somehow equivalent since they’re both trying to describe the same physical reality.

Let’s say that a model can be “more macroscopic” than another. Then chemistry is a “more macroscopic” description of physical reality than physics. They’re both ‘right’ but, being more macroscopic, Chemistry has higher expectations placed on it for computational tractability, and lower expectations for precision.

This suggests to me that we might want to consider a model that is “more macroscopic” than neural nets. They might be computationally intractable. Which brings me to my next and final point:

The elephant in the room is computational tractability. You said something during the party about “if the neural net is slow then we can just model the environment slowly.”

I think this is ignoring the fact that human intelligence is largely (if not entirely) written in terms of the structure of society. A human which grows up out of contact with society will never be “intelligent” in the same way that you or I are intelligent.

In fact, we do not have a single example anywhere in the universe of anything which we consider “intelligent” which has not had an absurd amount of personal contact with humans.

My belief is that human society is very very good at turning people into “intelligent” creatures, meaning creatures who can do art, who can learn math, who can play go, who can think about the mind…

The structures of society have evolved to allow us to reliably educate a sufficiently “normal” human to levels of intelligence heretofore unseen.

If we can’t connect to society a “creature” whose computational structure is a neural net, then I don’t think we’re going to be able to teach it, and I don’t think that it will be “intelligent” in the same way that you and I are intelligent.

One Response

Subscribe to comments with RSS.

  1. blim mickey said, on November 17, 2011 at 7:17 pm

    Read this last night just after catching up with the Oaf; could not help but share, as it contains similar contours at differing granularities.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: