For your humble average computational neuroscientist scrapping for a PhD, there are scant moments of reflection about the big picture, the bally-hoo, the why-we-spend-time-on-the-what-we-spend-so-much-freaking-time-doing. This disease can grow especially acute for the computationalist, in so many ways removed from the thing he/she simulates. So, I’d like to take a trip back to 1972 for my benefit (and maybe yours), when Charlie Gross and his lab at Princeton accidentally stumbled on something new and exciting, about a brain area considered `off-limits’ by some in the neuroscience establishment: inferotemporal cortex (IT, the thing I am currently embroiled in modeling). (Disclaimer: in case it wasn’t obvious, given that I was born in 1984, this particular nugget was acquired second-hand).

Much of the formative history of neuroscience, especially with regard to vision, concerns the `earliest’ bits of the visual pathway: the optics of the eye, retinal ganglion and geniculate cells, and the first stop in cortex–V1, or primary visual cortex. Since many neuroscientists fashion themselves reverse-engineers of the brain, it certainly made sense to begin with where they thought light impinging on the retina was first processed and relayed. This focus on the early areas also seemed to stem from the success of Hubel and Wiesel, who won the Nobel Prize for a large body of work on the structure of V1 and the properties of V1 cells. More precisely, they found that V1 cells in cat and monkey fired most vigorously to oriented bars in particular parts of the visual field, and that these orientation-tuned cells clumped together into processing units dubbed hypercolumns, which contained cells of all orientations fixed on a single part of the visual field. People developed this work both physiologically, by verifying and corroborating their work, and mathematically, by explaining how cells behind the retina `mapped’ to V1 and what sorts of equations describe orientation tuning. But, little was known about the areas further along in visual cortex, especially that pesky area IT.

Instead of proceeding methodically from V1 to V2 to V3 to V4 to posterior and then anterior IT, Charles Gross, for some mystical reason, decided to probe that cortical area situated near the bank of the superior temporal sulcus–IT–a brain area known at the time to mediate associative memory, not vision. If anything, IT did not respond to oriented bars and brief flashes of light, the sort of stuff that retinal ganglion, geniculate, and V1 cells happily lit up for. So, one might presume that IT was an area for the memory researchers, conceptually and methodologically blockaded from the vision people. Temporal lobe be damned, Gross plunked an electrode in IT using macaques and came upon the sort of happy accident one rarely hears of nowadays: while waving around all sorts of basic visual stimuli like oriented bars and squares, someone waved their hand in front of the monkey. Strangely enough, the cell discharged for the hand, leading the eminent researchers to create a set of stimuli with basic objects like monkey and human hands and pinwheels. This announced IT’s participation, along with the occipital lobe, in processing visual input.

Since that seminal work, it’s hard to say how far we’ve come. Some propose that IT has a columnar structure, with similar shapes, like circles of different radii for instance, exciting the same chunk of brain matter from the pia (top of cortex) to the white matter (bottom). Others claim that IT cells preferentially discharge for objects in depth, e.g. a sphere as opposed to a circle, yet an earlier study quantified IT cells’ selectivity for curvature using Fourier descriptors (2-D silhouettes that look like pinwheels). Labs, for instance Gross’, demonstrated that IT cells have enormous receptive fields, 12-15 degrees of visual angle, to relatively small ones, about 5 degrees. A few researchers claim that IT cell populations fire in stereotypical patterns that stay constant over time, while others say that individual IT cells are surprisingly plastic (i.e. change their firing pattern in response to certain inputs). Some even purport that IT cells act as a visual memory store, maintaining selectivity to stimuli even after they are no longer in the visual field. Most recently, DiCarlo and Cox suggested that IT detangles n-dimensional manifolds, where each dimension corresponds to the firing rate of a particular neuron, from V1 to IT. It’s safe to say that the neuroscience community is a series of islands, with scant ferries between them to share information and construct unifying functional theories. Most labs would agree that IT cells fire when a `complex object’ is in their receptive field, but even what a complex object or receptive field is remain under hot debate. Basically, we’re a ways from nailing down what IT does, aside from perhaps enabling visual object recognition.

Given the disarray in the experimental literature, it’s surprising to find anyone brave enough to construct a model. Some of the more prominent attempts include Wallis and Rolls’ work, which leveraged trace learning to form model cells that preferred the same object across views; that is, the trace allowed a cell to learn multiple views over a certain time window, say hundreds of milliseconds. Riesenhuber and Poggio conceived of IT response profiles as the outcome of local max operations, later extended by Serre et al., which also bears a resemblance to Fukushima’s Neocognitron. A lot of these models seem to draw from the idea of a Gaussian pyramid to create a compressed code for an image. In essence, the pyramid involves a series of Gaussian convolutions at larger and larger scales followed by a subsampling–taking an average of neighborhoods of pixels to form a smaller image, e.g. averaging 10×10 pixel windows in a 100×100 pixel image.

Gross’ original work inspired a great deal of research in IT, and now we’re left with a rather voluminous set of data. The community has certainly corroborated and expanded on the notion of IT as a cortical recognition area. The big question, still, is how IT does recognition (if that’s all it does). As with Hubel and Wiesel’s breakthrough with V1, one might say that we need to find the critical trigger feature, the thing or set of things that most if not all of IT cells process. Rigorous anatomy from prefrontal, parietal and other temporal areas that interconnect with IT should provide the other set of constraints that allows modelers to find the commonalities in the data necessary to simulate it with a single set of equations. If anything, at least there’s still plenty of work to go around!

(Image from Flickr user bob.bachand)