Differential reliance on sensory reinstatement and internally transformed representation during vivid retrieval of visual and auditory episodes

Auditory memory is considered less detailed yet more durable than visual memory, implying a modality-specific memory retrieval process. We used fMRI and multivoxel pattern analyses to examine how 25 participants encoded and retrieved naturalistic sounds and videos. Both auditory and visual targets reinstated item-specific fine activation patterns in the association cortex during retrieval, and reinstatement strength correlates with subjective memory vividness. However, auditory episodes showed a markedly larger reliance on internally constructed representations than visual episodes, quantified by retrieval-retrieval similarity after removing encoding traces. Sensory reinstatement correlated more to the (detail-related) posterior hippocampus, while internal representations also correlated to the (gist-related) anterior hippocampus. Furthermore, temporal voice areas preserved gist-level (human versus non-human) information from encoding to retrieval, whereas fusiform face representations degraded. These findings reveal that auditory and visual memories share a common sensory reinstatement mechanism, but differ in the neural mechanism that supports retrieval, with participants favouring gist over perceptual details during auditory memory retrieval.