Encoding manifolds constructed from grating responses organize responses to natural scenes in cortical visual areas

A challenge in sensory neuroscience is understanding how populations of neurons operate in concert to represent diverse stimuli. To meet this challenge, we have created "encoding manifolds" that reveal the overall responses of brain areas to diverse stimuli and organize individual neurons according to their selectivity and response dynamics. Here we use encoding manifolds to compare the population-level encoding of primary visual cortex (VISp) with that of five higher visual areas (VISam, VISal, VISpm, VISlm, and VISrl), using data from the Allen Institute Visual Coding-Neuropixels dataset from the mouse. We show that the topology of the encoding manifold for VISp and for higher visual areas is continuous, with smooth coordinates along which stimulus selectivity and response dynamics are organized with layer and cell-type specificity. Surprisingly, the manifolds revealed novel relationships between how natural scenes are encoded relative to static gratings--a relationship conserved across visual areas. Namely, neurons preferring natural scenes preferred either low or high spatial frequency gratings, but not intermediate ones. Analyzing responses by cortical layer reveals a preference for gratings concentrated in layer 6, whereas preferences for natural scenes tended to be higher in layers 2/3 and 4. The results demonstrate how machine learning approaches can be used to organize and visualize the structure of sensory coding, thereby revealing novel relationships within and across brain areas and sensory stimuli. Significance StatementManifolds have become commonplace for analyzing and visualizing neural responses. However, prior work has focused on building manifolds that organize diverse stimuli in neural response coordinates. Here, we demonstrate the utility of an alternative approach: building manifolds to represent neurons in stimulus/response coordinates, which we term encoding manifolds. This approach has several advantages, such as being able to directly visualize and compare how different brain areas encode diverse stimulus ensembles. This approach reveals novel relationships between layer-specific responses and the encoding of natural versus artificial stimuli.