BERTology Meets Biology: Interpreting Attention in Protein Language Models
Vig, J.; Madani, A.; Varshney, L. R.; Xiong, C.; Socher, R.; Rajani, N. F.
Show abstract
Transformer architectures have proven to learn useful representations for protein classification and generation tasks. However, these representations present challenges in interpretability. Through the lens of attention, we analyze the inner workings of the Transformer and explore how the model discerns structural and functional properties of proteins. We show that attention (1) captures the folding structure of proteins, connecting amino acids that are far apart in the underlying sequence, but spatially close in the three-dimensional structure, (2) targets binding sites, a key functional component of proteins, and (3) focuses on progressively more complex biophysical properties with increasing layer depth. We also present a three-dimensional visualization of the interaction between attention and protein structure. Our findings align with known biological processes and provide a tool to aid discovery in protein engineering and synthetic biology. The code for visualization and analysis is available at https://github.com/salesforce/provis.
Matching journals
The top 7 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- Paying Attention to Attention: High Attention Sites as Indicators of Protein Family and Function in Language Models 96%
- PandoGen: Generating complete instances of future SARS-CoV-2 sequences using Deep Learning 96%
- Predicting Affinity Through Homology (PATH): Interpretable Binding Affinity Prediction with Persistent Homology 96%
Similar papers in this journal
Similar papers in this journal
- Critiquing Protein Family Classification Models Using Sufficient Input Subsets 96%
- Combined topological data analysis and geometric deep learning reveal niches by the quantification of protein binding pockets 94%
- Enabling inference for context-dependent models of mutation by bounding the propagation of dependency 94%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.