When word order matters: human brains represent sentence meaning differently from large language models
Fodor, J.; Murawski, C.; Suzuki, S.
Show abstract
Large language models based on the transformer architecture are now capable of producing human-like language. But do they encode and process linguistic meaning in a human-like way? Here, we address this question by analysing 7T fMRI data from 30 participants reading 108 sentences each. These sentences are carefully designed to disentangle sentence structure from word meaning, thereby testing whether transformers representations of sentence meaning resemble those formed by the brain. We found that while transformer models match brain representations better than models that completely ignore word order, all transformer models performed poorly overall. Further, transformers were significantly inferior to models explicitly designed to encode the structural relations between words. Our results provide insight into the nature of sentence representation in the brain, highlighting the critical role of sentence structure. They also cast doubt on the claim that transformers represent sentence meaning similarly to the human brain.
Matching journals
The top 5 journals account for 50% of the predicted probability mass.
Similar papers in this journal
Similar papers in this journal
Similar papers in this journal
- Driving and suppressing the human language network using large language models 95%
- White matter connections of human ventral temporal cortex are organized by cytoarchitecture, eccentricity, and category-selectivity from birth 94%
- A brain-based universal measure of attention: predicting task-general and task-specific attention performance and their underlying neural mechanisms from task and resting state fMRI 94%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.