Predicting brain activity using Transformers
Adeli, H.; Minni, S.; Kriegeskorte, N.
Show abstract
The Algonauts challenge [Gifford et al., 2023] called on the community to provide novel solutions for predicting brain activity of humans viewing natural scenes. This report provides an overview and technical details of our submitted solution. We use a general transformer encoder-decoder model to map images to fMRI responses. The encoder model is a vision transformer trained using self-supervised methods (DINOv2). The decoder uses queries corresponding to different brain regions of interests (ROI) in different hemispheres to gather relevant information from the encoder output for predicting neural activity in each ROI. The output tokens from the decoder are then linearly mapped to the fMRI activity. The predictive success (challenge score: 63.5229, rank 2) suggests that features from self-supervised transformers may deserve consideration as models of human visual brain representations and shows the effectiveness of transformer mechanisms (self and cross-attention) to learn the mapping from features to brain responses. Code is available in this github repository.
Matching journals
The top 6 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- Alignment massive of auditory individual artificial networks with fMRI brain data leads to generalizable improvements in brain encoding and downstream tasks 96%
- Encoding neural representations of time-continuous stimulus-response transformations in the human brain with advanced deep neural networks 95%
- The Individualized Neural Tuning Model: Precise and generalizable cartography of functional architecture in individual brains 94%
Similar papers in this journal
- A Deep Graph Neural Network Architecture for Modelling Spatio-temporal Dynamics in resting-state functional MRI Data 94%
- DeepResBat: deep residual batch harmonization accounting for covariate distribution differences 94%
- Multi-dynamic Modelling Reveals Strongly Time-varying Resting fMRI Correlations 94%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.