Back

An Expert-Curated Benchmark for Frontier and Local LLMs Querying a Biomedical Knowledge Graph

Medagbe, J. F. P.; Kilyungi, B. M.; Isaac, A.; Kabua, A.; Lisso, F.; Wambua, S.; Williams, R. W.; Prins, P.

2026-10-11 bioinformatics
10.64898/2026.10.04.756532 bioRxiv
Show abstract

We combine GeneNetwork's 1TB structured database with its metadata as an RDF graph and an associated literature corpus. We benchmark frontier and local LLMs querying it through four architectures: document, graph and agentic retrieval-augmented generation (RAG), and a combination of the three. We evaluate them on 40 domain specific expert-curated questions with a new metric: the satisfaction frequency psi (psi, i.e, the share of questions answered with precision and recall of at least 70% or an f1 of at least 60%). psi ranks systems the same way precision, recall and f1 -- a machine learning metric built from the harmonic mean of precision and recall -- but is easier to read and assess for humans. System architecture had a large effect: with full use of Claude Opus 4.8 the plain document RAG of our earlier GNQA system had a psi of 36% against 86% for our new combined tools. Delegating most of the work from Claude Opus 4.8 to a local model halved query costs while still reaching a psi of 59%.

Matching journals

The top 6 journals account for 50% of the predicted probability mass.

1
Bioinformatics
◐ 1204 papers in training set
Top 1%
18.0%
Similar papers in this journal
2
Nature Communications
○ 5641 papers in training set
Top 20%
9.4%
Similar papers in this journal
3
GigaScience
◐ 212 papers in training set
Top 0.3%
7.7%
Similar papers in this journal
4
Database
◐ 61 papers in training set
Top 0.1%
6.1%
Similar papers in this journal
5
PLOS ONE
● 5266 papers in training set
Top 30%
4.7%
Similar papers in this journal
6
Journal of the American Medical Informatics Association
◐ 71 papers in training set
Top 0.8%
4.2%
50% of probability mass above
7
Bioinformatics Advances
◐ 203 papers in training set
Top 2%
3.4%
8
Nucleic Acids Research
◐ 1281 papers in training set
Top 5%
3.3%
9
Genome Biology
○ 637 papers in training set
Top 4%
3.1%
10
Scientific Reports
○ 3612 papers in training set
Top 35%
3.1%
11
Nature Methods
○ 385 papers in training set
Top 4%
2.4%
12
Nature Biotechnology
○ 172 papers in training set
Top 2%
2.3%
13
Scientific Data
○ 209 papers in training set
Top 1%
2.3%
14
Cell Systems
○ 201 papers in training set
Top 2%
2.0%
15
Patterns
○ 78 papers in training set
Top 1%
1.9%
16
PLOS Computational Biology
● 1863 papers in training set
Top 15%
1.6%
17
NAR Genomics and Bioinformatics
◐ 242 papers in training set
Top 3%
1.4%
18
BioData Mining
○ 22 papers in training set
Top 0.4%
1.1%
19
eLife
● 5828 papers in training set
Top 61%
1.1%
20
JMIR Medical Informatics
◐ 18 papers in training set
Top 0.7%
1.1%
21
BMC Bioinformatics
○ 457 papers in training set
Top 5%
1.0%
22
iScience
○ 1154 papers in training set
Top 32%
1.0%
23
JCO Clinical Cancer Informatics
● 22 papers in training set
Top 0.7%
1.0%
24
npj Digital Medicine
○ 118 papers in training set
Top 4%
0.8%
25
Computational and Structural Biotechnology Journal
● 242 papers in training set
Top 7%
0.8%
26
IEEE Transactions on Computational Biology and Bioinformatics
● 20 papers in training set
Top 0.6%
0.8%
27
GENETICS
◐ 483 papers in training set
Top 5%
0.6%
28
BMC Medical Informatics and Decision Making
○ 43 papers in training set
Top 2%
0.6%
29
Journal of Biomedical Informatics
○ 47 papers in training set
Top 1%
0.6%

"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.