Two cortical mechanisms for natural audiovisual processing

Understanding how the human brain processes natural audiovisual information remains a central challenge in cognitive neuroscience. Progress has been limited by the difficulty of modeling complex audiovisual stimuli - most prior work has therefore relied on short, controlled stimuli, or on stimuli from one modality at a time, leaving cortical mechanisms that support real-world comprehension poorly characterized. Further, while recent advances in artificial intelligence now enable the extraction of high-dimensional, time-resolved features from naturalistic stimuli, how cortical regions dynamically process and prioritize auditory and visual information as time unfolds remains largely unexplored. Using large-scale fMRI data collected while participants watched movies, we developed two complementary computational approaches relying on prediction performance to map the moment-by-moment dynamics of sensory processing across cortical regions: one detects sustained periods when one modality predicts a region substantially better than the other, identifying regions that switch the modality they encode for meaningful stretches of time; while the other identifies periods when both modalities predict the region well, revealing regions maintaining balanced representation of both auditory and visual information. Together, these analyses reveal two types of audiovisual processing across the cortex: a pair of "bows" that switch modalities (one posterior bow encircling category-selective visual cortex and another anterior bow spanning dorso-lateral frontal areas) and an arrow-like axis of regions that jointly represents both modalities (extending from lateral occipital cortex into the temporal cortex). The coexistence of these systems points to a cortical architecture that adaptively reweights sensory inputs while maintaining balanced multimodal representations, supporting robust comprehension of complex natural events.