Resolving a paradox about how vision is transformed into familiarity

While humans and other primates are generally good at remembering the images they have seen, they systematically remember some images better than others ("image memorability"). Here, we leverage image memorability to resolve a puzzle around how the brain transforms seeing into familiarity. Namely, the neural signal driving familiarity reports is thought to be repetition suppression, an overall reduction in the vigor of the population response in brain regions, including inferotemporal cortex (ITC). However, within ITC, more memorable images evoke higher firing rates than less memorable ones, even when they are repeated. These two observations conflict: if reduced firing leads to stronger memory signaling, then why are the images that induce greater firing more memorable? To investigate this "familiarity-memorability paradox", we compared neural activity in ITC and the hippocampus (HC) as two macaque monkeys performed a single-exposure image familiarity task. We found a resolution to the paradox, reflected in two pieces of evidence. First, memory and memorability were not completely overlapping in ITC, suggesting that familiarity signals could be extracted independently of memorability modulation. Second, responses in HC reflected a transformation of ITC responses that extracted memory, leaving behind stimulus-evoked firing modulation. This transformation could be accounted for by a linear decoder that selectively extracts memory and preserves larger ITC memory signals for more memorable images. These results resolve the familiarity-memorability paradox by demonstrating that ITC familiarity signals are a component (but not the entirety) of repetition suppression, and they establish the existence of a previously undescribed medial temporal lobe computation. SignificanceNeural representations of image memorability create a paradox: familiarity reports are thought to derive from reduced firing in inferotemporal cortex (ITC), but more memorable images evoke more vigorous responses there. To resolve this contradiction, we compared ITC with an area downstream, the hippocampus (HC), as macaque monkeys performed a single-exposure familiarity task. We found evidence for a computation between ITC and HC that selectively extracts ITC memory signals. This transformation could be explained by a linear decoder that preserves larger memory signals for more memorable images. This work resolves the familiarity-memorability paradox by showing that ITC memory signals have been misunderstood, and it reveals a previously undescribed role for the medial temporal lobe in familiarity behavior.