More than words: The online orchestration of word predictability, prosody, gesture, and mouth movements during natural language comprehension
Zhang, Y.; Frassinelli, D.; Tuomainen, J.; Skipper, J. I.; Vigliocco, G.
Show abstract
The ecology of human language is face-to-face interaction, comprising cues, like prosody, cospeech gestures, and mouth movements. Yet, the multimodal context is usually stripped away in experiments as dominant paradigms focus on linguistic processing only. In two studies we presented video-clips of an actress producing naturalistic passages to participants whose electroencephalographic activity was recorded. We quantified each cue and determined their effect on a well-established electroencephalographic marker of cognitive load in comprehension (N400). We found that brain responses to words were affected by informativeness of co-occurring multimodal cues, indicating that comprehension relies on linguistic and non-linguistic cues. Moreover, brain responses were affected by interactions between the multimodal cues, indicating that the impact of each cue dynamically changes based on the informativeness of other available cues. Thus, results show that multimodal cues are integral to comprehension, hence, our theories must move beyond the limited focus on speech and linguistic processing.
Matching journals
The top 8 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- The Interaction of Context Constraints and Predictive Validity during Sentence Reading 97%
- Error-based implicit learning in language: the effect of sentence context and constraint in a repetition paradigm 97%
- Prior context and individual alpha frequency (IAF) influence predictive processing during language comprehension 97%
Similar papers in this journal
- Neural markers of speech comprehension: measuring EEG tracking of linguistic speech representations, controlling the speech acoustics 97%
- Attentional engagement with target and distractor streams predicts speech comprehension in multitalker environments 97%
- Neural evidence for the prediction of animacy features during language comprehension: Evidence from MEG and EEG Representational Similarity Analysis 97%
Similar papers in this journal
- More than Words: Neurophysiological Correlates of Semantic Dissimilarity Depend on Comprehension of the Speech Narrative 97%
- The N400 ERP component reflects a learning signal during language comprehension 97%
- Semantic Reversal Anomalies under the Microscope: Task and Modality Influences on Language-Associated Event-Related Potentials 96%
Similar papers in this journal
- Parafoveal words can modulate sentence meaning: Electrophysiological evidence from an RSVP-with-flanker task 96%
- Parafoveal and foveal N400 effects in natural sentence reading: A timeline of semantic processing from fixation-related potentials 96%
- Word stress representations are language specific: evidence from event-related brain potentials 96%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.