Back

EvidenceTriangulator: A Large Language Model Approach to Synthesizing Causal Evidence across Study Designs

Shi, X.; Zhao, W.; Yang, C.; Du, J.

2024-03-19 health informatics
10.1101/2024.03.18.24304457 medRxiv
Show abstract

Health strategies increasingly emphasize both behavioral and biomedical interventions, yet the complex and often contradictory guidance on diet, behavior, and health outcomes complicates evidence-based decision-making. Evidence triangulation across diverse study designs is essential for establishing causality, but scalable, automated methods for achieving this are lacking. In this study, we assess the performance of large language models (LLMs) in extracting both ontological and methodological information from scientific literature to automate evidence triangulation. A two-step extraction approach--focusing on cause-effect concepts first, followed by relation extraction--outperformed a one-step method, particularly in identifying effect direction and statistical significance. Using salt intake and blood pressure as a case study, we calculated the Convergeny of Evidence (CoE) and Level of Evidence (LoE), finding a trending excitatory effect of salt on hypertension risk, with a moderate LoE. This approach complements traditional meta-analyses by integrating evidence across study designs, thereby facilitating more comprehensive assessments of public health recommendations.

Matching journals

The top 3 journals account for 50% of the predicted probability mass.

1
Journal of the American Medical Informatics Association
◐ 71 papers in training set
Top 0.1%
23.0%
Similar papers in this journal
2
Journal of Biomedical Informatics
○ 47 papers in training set
Top 0.1%
19.4%
Similar papers in this journal
3
PLOS ONE
● 5266 papers in training set
Top 17%
10.3%
Similar papers in this journal
50% of probability mass above
4
BMC Medical Research Methodology
○ 47 papers in training set
Top 0.1%
8.3%
Similar papers in this journal
5
BMC Medical Informatics and Decision Making
○ 43 papers in training set
Top 0.3%
6.6%
Similar papers in this journal
6
PLOS Digital Health
● 106 papers in training set
Top 1%
5.1%
7
Research Synthesis Methods
○ 20 papers in training set
Top 0.1%
3.4%
8
Journal of Clinical Epidemiology
○ 31 papers in training set
Top 0.3%
2.5%
9
BMJ Open
● 601 papers in training set
Top 9%
1.8%
10
JAMIA Open
◐ 42 papers in training set
Top 1%
1.4%
11
JMIR Medical Informatics
◐ 18 papers in training set
Top 0.6%
1.1%
12
Computers in Biology and Medicine
○ 128 papers in training set
Top 4%
1.1%
13
JMIR Research Protocols
◐ 21 papers in training set
Top 0.9%
1.1%
14
Healthcare
○ 17 papers in training set
Top 0.7%
1.0%
15
Neuroscience & Biobehavioral Reviews
○ 43 papers in training set
Top 0.6%
0.9%
16
International Journal of Medical Informatics
○ 26 papers in training set
Top 1%
0.6%
17
Contemporary Clinical Trials Communications
○ 11 papers in training set
Top 0.6%
0.5%
18
Journal of Medical Internet Research
◐ 87 papers in training set
Top 3%
0.5%
19
Bioinformatics
◐ 1204 papers in training set
Top 10%
0.5%
20
Database
◐ 61 papers in training set
Top 1%
0.5%
21
Systematic Reviews
○ 15 papers in training set
Top 0.6%
0.5%
22
Medical Decision Making
○ 12 papers in training set
Top 0.5%
0.5%
23
International Journal of Environmental Research and Public Health
○ 128 papers in training set
Top 6%
0.5%
24
Value in Health
○ 11 papers in training set
Top 0.3%
0.5%
25
Scientific Reports
○ 3612 papers in training set
Top 79%
0.5%

"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.