Back

Show Your Work: Verbatim Evidence Requirements and Automated Assessment for Large Language Models in Biomedical Text Processing

2026-03-04 health informatics Title + abstract only
View on medRxiv
Show abstract

PurposeLarge language models (LLMs) are used for biomedical text processing, but individual decisions are often hard to audit. We evaluated whether enforcing a mechanically checkable "show your work" quote affects accuracy, stability, and verifiability for trial eligibility-scope classification from abstracts. MethodsWe used 200 oncology randomized controlled trials (2005 - 2023) and provided models with only the title and abstract. Trials were labeled with whether they allowed for the inclusio...

Predicted journal destinations

1
Journal of the American Medical Informatics Association
53 training papers
#1 23.7%
2
Journal of Biomedical Informatics
37 training papers
Top 1% 6.8%
3
npj Digital Medicine
85 training papers
Top 3% 6.8%
4
JAMIA Open
35 training papers
Top 1% 6.8%
5
BMC Medical Informatics and Decision Making
36 training papers
Top 0.8% 6.8%
6
PLOS Digital Health
88 training papers
Top 4% 6.0%
7
Journal of Medical Internet Research
81 training papers
Top 4% 5.4%
8
BMC Medical Research Methodology
41 training papers
Top 0.4% 4.1%
9
Nature Communications
483 training papers
Top 26% 4.1%
10
Scientific Reports
701 training papers
Top 64% 4.1%
11
PLOS ONE
1737 training papers
Top 91% 2.1%
12
International Journal of Medical Informatics
25 training papers
Top 3% 2.1%
13
JMIR Medical Informatics
16 training papers
Top 1% 2.1%
14
Computers in Biology and Medicine
39 training papers
Top 6% 1.4%
15
BMJ Open
553 training papers
Top 56% 1.4%
16
Frontiers in Digital Health
18 training papers
Top 0.8% 1.4%
17
JAMA Network Open
125 training papers
Top 19% 1.2%
18
Journal of Clinical Epidemiology
29 training papers
Top 2% 1.0%
19
Cancers
57 training papers
Top 8% 0.8%
20
Nature Medicine
88 training papers
Top 16% 0.8%
21
Patterns
15 training papers
Top 3% 0.8%
22
The Lancet Digital Health
25 training papers
Top 2% 0.8%
23
Bioinformatics
24 training papers
Top 3% 0.8%
24
Communications Medicine
63 training papers
Top 6% 0.8%
25
Scientific Data
30 training papers
Top 5% 0.5%