Limitations and Enhancements in Genomic Language Models: Dynamic Selection Approach
Qiu, S.
Show abstract
1Genomic Language Models (GLMs), which learn from nucleotide sequences, are crucial for understanding biological principles and excel in tasks such as sequence generation and classification. However, state-of-the-art models vary in training methods, architectures, and tokenization techniques, resulting in different strengths and weaknesses. We propose a multi-model fusion approach with a dynamic model selector that effectively integrates three models with distinct architectures. This fusion enhances predictive performance in downstream tasks, outperforming any individual model and achieving complementary advantages. Our comprehensive analysis reveals a strong correlation between model performance and motif prominence in sequences. Nevertheless, overreliance on motifs may limit the understanding of ultra-short core genes and the context of ultra-long sequences. Importantly, based on our in-depth experiments and analyses of the current three leading models, we identify unresolved issues and suggest potential future directions for the development of genomic models. The code, data, and pre-trained model are available at https://github.com/Jacob-S-Qiu/glm_dynamic_selection.
Matching journals
The top 5 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- Deep6mA: a deep learning framework for exploring similar patterns in DNA N6-methyladenine sites across different species 95%
- DeepHE: Accurately Predicting Human Essential Genes based on Deep Learning 95%
- Assessing the Performance of Methods for Cell Clustering from Single-cell DNA Sequencing Data 95%
Similar papers in this journal
Similar papers in this journal
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.