Efficient and accurate sequence generation with small-scale protein language models
Serrano, Y.; Roda, S.; Guallar, V.; Molina, A.
Show abstract
Large Language Models (LLMs) have demonstrated exceptional capabilities in understanding contextual relationships, outperforming traditional methodologies in downstream tasks such as text generation and sentence classification. This success has been mirrored in the realm of protein language models (pLMs), where proteins are encoded as text via their amino acid sequences. However, the training of pLMs, which involves tens to hundreds of millions of sequences and hundreds of millions to billions of parameters, poses a significant computational challenge. In this study, we introduce a Small-Scale Protein Language Model (SS-pLM), a more accessible approach that requires training on merely millions of representative sequences, reducing the number of trainable parameters to 14.8M. This model significantly reduces the computational load, thereby democratizing the use of foundational models in protein studies. We demonstrate that the performance of our model, when fine-tuned to a specific set of sequences for generation, is comparable to that of larger, more computationally demanding pLM.
Matching journals
The top 4 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- Controllable Protein Design via Autoregressive Direct Coupling Analysis Conditioned on Principal Components 96%
- Predicting changes in protein thermodynamic stability upon point mutation with deep 3D convolutional neural networks 95%
- Computational design of novel Cas9 PAM-interacting domains using evolution-based modelling and structural quality assessment 95%
Similar papers in this journal
Similar papers in this journal
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.