The geometry of hidden representations of protein language models
Valeriani, L.; Cuturello, F.; Ansuini, A.; Cazzaniga, A.
Show abstract
Protein language models (pLMs) transform their input into a sequence of hidden representations whose geometric behavior changes across layers. Looking at fundamental geometric properties such as the intrinsic dimension and the neighbor composition of these representations, we observe that these changes highlight a pattern characterized by three distinct phases. This phenomenon emerges across many models trained on diverse datasets, thus revealing a general computational strategy learned by pLMs to reconstruct missing parts of the data. These analyses show the existence of low-dimensional maps that encode evolutionary and biological properties such as remote homology and structural information. Our geometric approach sets the foundations for future systematic attempts to understand the space of protein sequences with representation learning techniques.
Matching journals
The top 8 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- EGRET: Edge Aggregated Graph Attention Networks and Transfer Learning Improve Protein-Protein Interaction Site Prediction 95%
- PRIEST - Predicting viral mutations with immune escape capability of SARS-CoV-2 using temporal evolutionary information 95%
- An Analysis of Protein Language Model Embeddings for Fold Prediction 94%
Similar papers in this journal
- Interpretable Pairwise Distillations for Generative Protein Sequence Models 96%
- Learning massive interpretable gene regulatory networks of the human brain by merging Bayesian Networks 95%
- Predicting Affinity Through Homology (PATH): Interpretable Binding Affinity Prediction with Persistent Homology 95%
Similar papers in this journal
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.