Convergence of efficient and predictive coding in multimodal sensory processing

The existence of pathways connecting different sensory modalities in the brain challenges the traditional view of sensory systems as operating independently. However, the reasons and mechanisms underlying these interactions remain largely unknown, and no computational framework currently addresses these questions. We propose a theory of sensory processing in canonical circuits - networks of excitatory and inhibitory neurons ubiquitously found in the brain. Our theory incorporates cross-modal feedback and demonstrates that these networks can orchestrate precise mathematical computations through distinct circuit components. These computations generate neural codes that are simultaneously efficient and predictive, unifying two classical coding schemes. Our framework treats unimodal processing as a special case and accounts for olfactory-visual and auditory-somatosensory interactions observed in experiments, while offering new predictions about the role of neural feedback in optimizing multimodal codes. By bridging normative theories of sensory coding, this study provides insights into the principles governing interactions between the senses.