On Intelligence, by Jeff Hawkins (2004)

What a book. I don’t know where to start.

I had hoped to come away with an understanding of the overall theory and had expected that the details would be mostly irrelevant to me as soon as they had served their purpose of explaining some higher-level concept, but fuck me, the details are interesting.

The way the problem is broken down makes me desperately want to understand more of it. I know nothing about neuroscience, but the whole book reads like an invitation; here’s the goal, and here’s what we know, connect the two.

A few things stood out as excellent while reading; clearly drawn boundaries between established science and the author’s thinking, elegant abstractions of the physical machinery of the brain, and attempts to answer nearly every question I thought of as I worked through it.

I can’t wait to read the follow up, A Thousand Brains, which is said to have solutions to some of the problems introduced in this book.

Useful definitions

• Neurons are nerve cells that communicate with other cells via a spike, or action potential, which is a brief electrical pulse.

• Neurons have wirelike structures called axons and dendrites. When an axon from one neuron touches the dendrite of another, it forms a synapse.

• Synapses are a connection between two cells which allow the spike of one to influence the other. The influence can be inhibitory (a signal in one makes the other less likely to signal) or excitatory (a signal in one makes the other more likely to signal).

• The brain remembers things by creating invariant representations. These are stable internal representations of concepts or objects independent of the details, for example, you recognise faces regardless of the orientation, position of facial features, etc., because the invariant representation you have for “face” does not depend directly on these variable details.

Some very rough notes

The neocortex is the outermost layer of the brain. It is 2mm thick, has six layers, no discernable separation between different areas, and it houses intelligence.

In 1978 Vernon Mountcastle suggested that because the neocortex looks the same throughout, it could all be doing the same thing. Regardless of whether it’s vision, hearing, touch, etc., it could be doing the same processing with different sensory inputs. A common cortical algorithm. The book runs with this assumption.

The neocortex remembers patterns by storing invariant representations of them. Taking vision as an example, whenever a face is anywhere in your view, a certain set of cells in the visual part of your cortex lights up. It doesn’t matter who’s face it is or what it looks like, or even if it’s half of face instead of a full one, as soon as a pattern partially matching that of your internal representation of a face is present in your line of sight, those cells become active and stay active until the face disappears from view.

This happens automatically. You don’t need to dig around in your brain for the memory of what a face looks like and match it to the sensory inputs you’re receiving; it activates automatically as the result of a face being in view.

These representations have a temporal dimension as well as a spatial one. You can’t recognise a song from a single beat for the same reason that you can’t remember an entire song at once. Your invariant representation of it has a temporal dimension, it’s a pattern that plays out over time.

Figure 1 depicts the visual pathway in the neocortex on the left.

There are four hierarchical levels in the visual pathway; V1, V2, V4, and IT. V1 is at the bottom and receives sensory input first. There is no physical basis for the hierarchy (no “above” or “below” in the brain) other than the manner in which the regions communicate to each other, although lower regions are much larger than higher regions.

The conventional view sees each level in the hierarchy as a distinct region performing a similar task throughout. V1 would perform low level recognition on its inputs, such as the detection of lines, then pass it up the to V2 which might in turn recognise shapes, and so on. Higher level regions, such as IT, are called “association” areas. These combine the information from lower regions and have a bird’s eye view over the pathway below. In this view, invariant representations are formed only when the input reaches the top, at IT.

The book challenges that traditional view by proposing that each traditional region (e.g. all of V2) should be divided into many smaller subregions, as shown on the left of Figure 1. Applying the common cortical algorithm assumption leads the book to propose that each of these subregions performs the same task as any other. That means that invariant representations exist in all of them, not just in the higher association areas like IT. Every region forms memories of patterns in the inputs coming from the regions below it.

Inside each sub-region (shown on the right of Figure 1), information flows up and down cortical columns. A cortical column represents the smallest unit of computation in the neocortex. Information in each cortical column behaves similarly, for example, converging inputs from lower regions arrive at Layer 4 before being passed up to Layers 2 and 3.

In the flow of information up and down columns and between regions, the book attempts to identify the mechanisms that enable the predictive activation of invariant representations that are necessary to explain our experience.

Taking imagination as an example. The book says that if you close your eyes and imagine a hippopotamus, the visual area of your brain associated with seeing a hippopotamus lights up. It then shows how the connections between neurons in a cortical column allow for this by turning its predictions into its own inputs, essentially feeding back into itself.

Highlights