Shared but temporally distinct neural representations support semantic matching across word and picture formats: evidence from EEG decoding and temporal generalization analyses

Audiovisual semantic matching can be achieved using either written words or pictures, yet whether these formats engage shared semantic matching representations with similar temporal dynamics remains unclear. We recorded electroencephalography from 27 participants while they performed audiovisual semantic matching tasks in which spoken words were paired with either written words or pictures. Stimuli included both natural and man-made objects. Time-resolved multivariate pattern analyses (MVPA or decoding), cross-decoding, and temporal generalization analyses were used to characterize the temporal dynamics of semantic processing. Reliable decoding of matching versus mismatching judgments emerged in both word and picture conditions. Decoding onset that significant above chance level occurred earlier for written words than for pictures and cross-decoding analyses revealed successful generalization between word and picture formats. Temporal generalization analyses further demonstrated distinct representational dynamics across formats, with word processing characterized by predominantly time-specific neural representations and picture processing showing more sustained and temporally stable representations. In addition, matching-related discrimination emerged earlier for natural objects than for man-made objects across both formats. The results suggest that speech-word matching shows earlier neural evidence of audiovisual alignment than speech-picture matching, potentially reflecting differences in how auditory linguistic input is integrated with visual information across representational formats.