A linguistic representation in the visual system underlies successful lipreading
Nidiffer, A. R.; Cao, C. Z.; O'Sullivan, A. E.; Lalor, E. C.
Show abstract
There is considerable debate over how visual speech is processed in the absence of sound and whether neural activity supporting lipreading occurs in visual brain areas. Surprisingly, much of this ambiguity stems from a lack of behaviorally grounded neurophysiological findings. To address this, we conducted an experiment in which human observers rehearsed audiovisual speech for the purpose of lipreading silent versions during testing. Using a combination of computational modeling, electroencephalography, and simultaneously recorded behavior, we show that the visual system produces its own specialized representation of speech that is 1) well-described by categorical linguistic units ("visemes") 2) dissociable from lip movements, and 3) predictive of lipreading ability. These findings contradict a long-held view that visual speech processing co-opts auditory cortex after early visual processing stages. Consistent with hierarchical accounts of visual and audiovisual speech perception, our findings show that visual cortex performs at least a basic level of linguistic processing.
Matching journals
The top 8 journals account for 50% of the predicted probability mass.
Similar papers in this journal
Similar papers in this journal
- Neural Tracking Measures of Speech Intelligibility: Manipulating Intelligibility while Keeping Acoustics Unchanged 96%
- Orthogonal neural codes for phonetic features in the infant brain 96%
- The neural signature of numerosity: Separating numerical and continuous magnitude extraction in visual cortex with frequency-tagged EEG 95%
Similar papers in this journal
- Audio-visual combination of syllables involves time-sensitive dynamics following from fusion failure 96%
- Predicting the subjective intensity of imagined sensory experiences from electrophysiological measures of oscillatory brain activity. 96%
- Bayesian Prior Uncertainty and Surprisal Elicit Distinct Neural Patterns During Sound Localization in Dynamic Environments 94%
Similar papers in this journal
- Rapid computations of spectrotemporal prediction error support perception of degraded speech 95%
- Low-frequency Neural Activity Reflects Rule-based Chunking during Speech Listening 95%
- Local cortical desynchronization and pupil-linked arousal differentially shape brain states for optimal sensory performance 95%
Similar papers in this journal
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.