Evo 2 as a classification machine: evidence from in-context learning and mechanistic interpretability
Bertolini Agnoletto, L.; Curion, F.; Petrillo, M.; Leoni, G.; Ronco, M.; Ruiz Serra, V.; Consoli, S.; Ceresa, M.
Show abstract
In-context learning (iCL) is an emergent capability of Large Language Models (LLMs), allowing them to perform new tasks at inference time using prompt-injected examples. While extensively studied in LLMs, the boundaries of its capabilities and underlying mechanisms remain poorly characterised in genomic Language Models (gLMs). Here, we map the operating regime of Evo 2, a nucleotide-level foundation gLM, across five binary classification tasks spanning biological and artificial sequences. We observe robust iCL on shorter natural sequences (F1=0.902 for miRNA, 0.785 for Toxins), degrading with sequence length, and collapsing at kilobase scale. Strikingly, we find no benefit in model scaling, as the 7B model systematically outperforms the 40B variant. We further find that perplexity, a widely used gLM performance proxy, poorly predicts accuracy. Mechanistic interpretability reconciles these observations: the logit-lens profiling suggests a prediction-generalisation trade-off, while Jacobian Scope indicates models might track prompts' structure rather than the signal-carrying content.
Matching journals
The top 7 journals account for 50% of the predicted probability mass.
Similar papers in this journal
Similar papers in this journal
- Correcting gradient-based interpretations of deep neural networks for genomics 94%
- EvoRMD: Integrating Biological Context and Evolutionary RNA Language Models for Interpretable Prediction of RNA Modifications 94%
- EvoAug: improving generalization and interpretability of genomic deep neural networks with evolution-inspired data augmentations 93%
Similar papers in this journal
Similar papers in this journal
- Gated recurrence enables simple and accurate sequence prediction in stochastic, changing, and structured environments 94%
- Deep learning linking mechanistic models to single-cell transcriptomics data reveals transcriptional bursting in response to DNA damage 93%
- Generative modeling for RNA splicing predictions and design 93%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.