Modality-Agnostic Decoding of Vision and Language from fMRI
Nikolaus, M.; Mozafari, M.; Berry, I.; Asher, N.; Reddy, L.; VanRullen, R.
Show abstract
Humans perform tasks involving the manipulation of inputs regardless of how these signals are perceived by the brain, thanks to representations that are invariant to the stimulus modality. In this paper, we present modality-agnostic decoders that leverage such modality-invariant representations to predict which stimulus a subject is seeing, irrespective of the modality in which the stimulus is presented. Training these modality-agnostic decoders is made possible thanks to our new large-scale fMRI dataset SemReps-8K, released publicly along with this paper. It comprises 6 subjects watching both images and short text descriptions of such images, as well as conditions during which the subjects were imagining visual scenes. We find that modality-agnostic decoders can perform as well as modality-specific decoders, and even outperform them when decoding captions and mental imagery. Further, a searchlight analysis revealed that large areas of the brain contain modality-invariant representations. Such areas are also particularly suitable for decoding visual scenes from the mental imagery condition.
Matching journals
The top 4 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- Understanding transformation tolerant visual object representations in the human brain and convolutional neural networks 96%
- Functional selectivity for naturalistic social interaction perception in the human superior temporal sulcus 96%
- Capturing Brain-Cognition Relationship: Integrating Task-Based fMRI Across Tasks Markedly Boosts Prediction and Test-Retest Reliability 96%
Similar papers in this journal
- Encoding neural representations of time-continuous stimulus-response transformations in the human brain with advanced deep neural networks 97%
- Representational similarity learning reveals a graded multi-dimensional semantic space in the human anterior temporal cortex 96%
- Identifying and characterizing scene representations relevant for categorization behavior 96%
Similar papers in this journal
Similar papers in this journal
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.