RNA-xLSTM: Evaluating xLSTM as an Alternative Foundation to Transformers in RNA Modeling
Pintaric, M.; Penic, R. J.; Sikic, M.
Show abstract
Transformer-based architectures currently achieve state-of-the-art performance across a wide range of domains, including biological sequence modeling. Motivated by the recent introduction of the xLSTM architecture, we investigate its effectiveness for RNA sequence modeling by comparing a 33.7M-parameter RNA-xLSTM model against two leading RNA language models: RNA-FM and RiNALMo-33M. We pretrain RNA-xLSTM on the RNAcentral database and evaluate its performance on two downstream tasks: RNA secondary structure prediction and splice site prediction. Our results show that while RNA-xLSTM underperforms compared to the similarly sized RiNALMo, it does outperform the larger RNA-FM model on certain tasks. However, its overall performance remains inconsistent, and its advantages over transformer-based models are unclear, suggesting that further work is needed to assess its true potential in RNA modeling.
Matching journals
The top 5 journals account for 50% of the predicted probability mass.
Similar papers in this journal
Similar papers in this journal
Similar papers in this journal
- Predicting RNA 3D structure and conformers using a pre-trained secondary structure model and structure-aware attention 95%
- Interpreting Neural Networks for Biological Sequences by Learning Stochastic Masks 94%
- Clair: Exploring the limit of using a deep neural network on pileup data for germline variant calling 93%
Similar papers in this journal
- De novo prediction of RNA-protein interactions with Graph Neural Networks 95%
- Deep Learning for RNA Secondary Structure Determination: Gauging Generalizability and Broadening the Scope of Traditional Methods 94%
- bpRNA-align: Improved RNA Secondary Structure Global Alignment for Comparing and Clustering RNA Structures 93%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.