Sparse Autoencoders Reveal Structural and Family-level Features in BiRNA-BERT
Hossain, M. S.; Sojib, M. R.; Tahmid, M. T.; Rahman, M. S.
Show abstract
Motivation: RNA language models learn representations that support structure and function prediction, but which biological concepts their hidden states encode remains unclear. Sparse autoencoders (SAEs) decompose hidden states into interpretable features, yet have not been applied to RNA language models, where byte-pair tokenization breaks the one-token-one-nucleotide correspondence that nucleotide-level attribution assumes. Results: We present SPIRAL, a layer-wise SAE analysis of BiRNA-BERT. Independent SAEs at layers 0, 5, and 11 expand each 768-dimensional hidden state into 6,144 features while preserving model behaviour (explained variance above 0.99997; masked-language-model sequence recovery near 99.7%). Tokenizer-aware offset propagation aligns features to nucleotides: at layer 5, 44.3% of tested features are significantly associated with bpRNA secondary-structure classes (mean enrichment 1.61x), and all 1,237 eligible features with RNAcentral RNA types. Sparse profiles raise k-nearest-neighbour balanced accuracy from 0.328 to 0.359 over dense embeddings at layer 5. Availability and Implementation: Source code is available at https://github.com/SadatHossain01/SPIRAL; the code, evaluation data, and trained SAE checkpoints are archived at https://doi.org/10.5281/zenodo.21891845. Contact: mrahman@cse.buet.ac.bd
Matching journals
The top 3 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- Graph neural representational learning of RNA secondary structures for predicting RNA-protein interactions 95%
- DUETT quantitatively identifies known and novel events in nascent RNA structural dynamics from chemical probing data 95%
- ConsAlign: simultaneous RNA structural aligner based on rich transfer learning and thermodynamic ensemble model of alignment scoring 94%
Similar papers in this journal
- Juggling offsets unlocks RNA-seq tools for fast scalable differential usage, aberrant splicing and expression analyses. 96%
- EvoRMD: Integrating Biological Context and Evolutionary RNA Language Models for Interpretable Prediction of RNA Modifications 96%
- HydraRNA: a hybrid architecture based full-length RNA language model 96%
Similar papers in this journal
- Needlestack: an ultra-sensitive variant caller for multi-sample next generation sequencing data 94%
- Informative RNA-base embedding for functional RNA structural alignment and clustering by deep representation learning 93%
- FASTCAR: Rapid alignment-free prediction of sequence alignment identity scores 93%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.