During natural vision, semantic novelty modulates fixation-related processing in primate cortex

We sample visual scenes with short gaze fixations separated by saccades. While low-level transsaccadic integration is known, semantic integration across multiple fixations remains unclear. We hypothesized that the brain predicts semantic information from one fixation to the next, and therefore postulated a neural signal associated with semantic novelty for each saccade. Novelty was measured using a deep network on foveal vision. Novelty modulated frontal and occipital fixation-related potentials in human EEG during natural viewing of full-length movies (3.4x106 saccades). Intracranial recordings in humans (9.0x104 saccades) and non-human primates (3.3x104 saccades) revealed broadband high-frequency activity modulations in ventromedial/dorsal visual streams and frontal brain areas. This modulation was stronger for movies than static images, and frontal modulation preceded occipital modulation, suggesting top-down effects. This ubiquitous modulation of fixation-related activity with novelty suggests that foveal representations are integrated across saccades in primates. Significance statementPrimate vision relies on eye movements to sample the world, yet how the brain integrates high-level meaning across these "snapshots" remains a mystery. By combining deep-learning models with massive electrophysiological datasets from humans and macaques, we show that the brain predicts the semantic content of upcoming fixations. We identified a "semantic novelty" signal that modulates neural activity across the primate cortex, appearing in frontal areas even before a saccade is completed. These findings reveal that visual processing is not a series of independent glimpses, but a continuous, predictive process that integrates foveal information across space and time. This work bridges biological vision and artificial intelligence, providing a new framework for how the brain maintains a coherent understanding of a dynamic world.