Neural Dynamics of Object Manifold Alignment in the Ventral Stream

Visual neurons respond across a vast landscape of images, comprising objects, textures, and places. Natural images can be parameterized using deep generative networks, raising the question of whether latent factors learned by some networks control images in ways that better align with visual neurons. We studied neurons in areas V1, V4 and posterior IT, optimizing images using a closed-loop evolutionary algorithm. We used two generative image models: (1) DeePSim, which parameterizes local image patterns, and (2) BigGAN which parameterizes object identity and nuisance variables. We found that neurons could guide image optimization on both pattern- and object-based image manifolds across areas; V1 aligned best with the DeePSim image space, whereas PIT aligned well with both DeePSim and BigGAN spaces. While initially PIT neurons responded well to the textural manifold, their responses to objects also emerged over time, suggesting that object-like responses required further processing. We identified similar local features common to both textural and object images, but not optimal global configuration. We conclude that visual cortex neurons are aligned to a representational space not yet captured by current artificial model of the visual system.