Back

GeneTuring tests GPT models in genomics

Hou, W.; Ji, Z.

2023-03-13 bioinformatics
10.1101/2023.03.11.532238 bioRxiv
Show abstract

Large language models (LLMs) show promise in biomedical research, but their effectiveness for genomic inquiry remains unclear. We developed GeneTuring, a benchmark consisting of 16 genomics tasks with 1,600 curated questions, and manually evaluated 48,000 answers from ten LLM configurations, including GPT-4o (via API, ChatGPT with web access, and a custom GPT setup), GPT-3.5, Claude 3.5, Gemini Advanced, GeneGPT (both slim and full), BioGPT, and BioMedLM. A custom GPT-4o configuration integrated with NCBI APIs, developed in this study as SeqSnap, achieved the best overall performance. GPT-4o with web access and GeneGPT demonstrated complementary strengths. Our findings highlight both the promise and current limitations of LLMs in genomics, and emphasize the value of combining LLMs with domain-specific tools for robust genomic intelligence. GeneTuring offers a key resource for benchmarking and improving LLMs in biomedical research. Biographical NoteDr. Wenpin Hou is an Assistant Professor (tenure-track) in the Department of Biostatistics at Columbia University and member of its Data Science Institute, developing AI and statistical methods for decoding gene regulatory programs from single-cell and spatial multiomics data.

Matching journals

The top 5 journals account for 50% of the predicted probability mass.

1
Bioinformatics Advances
◐ 203 papers in training set
Top 0.1%
14.8%
Similar papers in this journal
2
BMC Bioinformatics
○ 457 papers in training set
Top 0.4%
12.9%
Similar papers in this journal
3
Scientific Data
○ 209 papers in training set
Top 0.3%
7.9%
Similar papers in this journal
4
GigaScience
◐ 212 papers in training set
Top 0.2%
7.9%
Similar papers in this journal
5
Bioinformatics
◐ 1204 papers in training set
Top 3%
7.9%
Similar papers in this journal
50% of probability mass above
6
NAR Genomics and Bioinformatics
◐ 242 papers in training set
Top 0.5%
6.2%
7
Scientific Reports
○ 3612 papers in training set
Top 15%
5.5%
8
Genome Biology
○ 637 papers in training set
Top 3%
4.3%
9
Briefings in Bioinformatics
◐ 354 papers in training set
Top 3%
3.2%
10
Nucleic Acids Research
◐ 1281 papers in training set
Top 6%
3.2%
11
Database
◐ 61 papers in training set
Top 0.3%
2.6%
12
Computational and Structural Biotechnology Journal
● 242 papers in training set
Top 2%
2.4%
13
PLOS Computational Biology
● 1863 papers in training set
Top 14%
1.7%
14
Nature Communications
○ 5641 papers in training set
Top 44%
1.7%
15
PLOS ONE
● 5266 papers in training set
Top 50%
1.3%
16
F1000Research
○ 88 papers in training set
Top 2%
1.1%
17
JCO Clinical Cancer Informatics
● 22 papers in training set
Top 0.5%
1.1%
18
iScience
○ 1154 papers in training set
Top 30%
1.0%
19
Patterns
○ 78 papers in training set
Top 2%
1.0%
20
European Journal of Human Genetics
○ 58 papers in training set
Top 1%
0.8%
21
IEEE Transactions on Computational Biology and Bioinformatics
● 20 papers in training set
Top 0.6%
0.8%
22
BioData Mining
○ 22 papers in training set
Top 0.7%
0.8%
23
Nature Machine Intelligence
○ 70 papers in training set
Top 3%
0.6%
24
Journal of Bioinformatics and Systems Biology
○ 15 papers in training set
Top 0.4%
0.6%
25
BMC Genomics
○ 406 papers in training set
Top 9%
0.6%
26
eLife
● 5828 papers in training set
Top 70%
0.6%

"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.