Deep learning algorithms reveal a new visual-semantic representation of familiar faces in human perception and memory
Shoham, A.; Grosbard, I.; Patashnik, O.; Cohen-Or, D.; Yovel, G.
Show abstract
Recent studies show significant similarities between the representations humans and deep neural networks (DNNs) generate for faces. However, two critical aspects of human face recognition are overlooked by these networks. First, human face recognition is mostly concerned with familiar faces, which are encoded by visual and semantic information, while current DNNs solely rely on visual information. Second, humans represent familiar faces in memory, but representational similarities with DNNs were only investigated for human perception. To address this gap, we combined visual (VGG-16), visual-semantic (CLIP), and natural language processing (NLP) DNNs to predict human representations of familiar faces in perception and memory. The visual-semantic network substantially improved predictions beyond the visual network, revealing a new visual-semantic representation in human perception and memory. The NLP network further improved predictions of human representations in memory. Thus, a complete account of human face recognition should go beyond vision and incorporate visual-semantic, and semantic representations.
Matching journals
The top 6 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- DeepAction: A MATLAB toolbox for automated classification of animal behavior in video 94%
- Movement trajectories as a window into the dynamics of emerging neural representations. 93%
- Orthogonal Representations of Object Shape and Category in Deep Convolutional Neural Networks and Human Visual Cortex 93%
Similar papers in this journal
Similar papers in this journal
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.