Back

Trustworthiness of Systematic Review Automation - An interview at Coventry University

Jiang, X.

2023-12-15 health informatics
10.1101/2023.12.14.23299933 medRxiv
Show abstract

To reduce both time and labour for systematic review, much effort has been made on developing automated tools using artificial intelligence and machine learning in nearly two decades. Unfortunately, systematic review automation tools still face a serious problem of social acceptance. Previous studies identified lack of trust as one important adoption barrier for systematic review automation. However, further discussion about building trust was limited to building trusted evidence base by benchmarked large-scale evaluation. This study extended the previous discussions of the trustworthiness of systematic review automation. Through semi-structure interviews with regular systematic reviewers, we tried to not only get answers for to what extent human reviewers trust automated tools and why, but also reveal more measures of building trust from human reviewers points of view and the impact of such measures on the trust in and adoption of systematic review automation tools. We believe that the results of this study may also shed light on some new directions of systematic review automation research.

Matching journals

The top 6 journals account for 50% of the predicted probability mass.

1
Research Synthesis Methods
○ 20 papers in training set
Top 0.1%
12.7%
Similar papers in this journal
2
PLOS Digital Health
● 106 papers in training set
Top 0.5%
11.9%
Similar papers in this journal
3
Journal of the American Medical Informatics Association
◐ 71 papers in training set
Top 0.4%
7.9%
Similar papers in this journal
4
BMC Medical Research Methodology
○ 47 papers in training set
Top 0.1%
7.9%
Similar papers in this journal
5
BMJ Open
● 601 papers in training set
Top 4%
6.3%
Similar papers in this journal
6
DIGITAL HEALTH
○ 17 papers in training set
Top 0.1%
5.5%
50% of probability mass above
7
Neuroscience & Biobehavioral Reviews
○ 43 papers in training set
Top 0.1%
4.4%
8
PLOS ONE
● 5266 papers in training set
Top 31%
4.3%
9
Journal of Clinical Epidemiology
○ 31 papers in training set
Top 0.2%
3.2%
10
BMJ Health & Care Informatics
● 15 papers in training set
Top 0.4%
2.7%
11
Scientific Reports
○ 3612 papers in training set
Top 45%
2.1%
12
Systematic Reviews
○ 15 papers in training set
Top 0.2%
2.1%
13
Journal of Medical Internet Research
◐ 87 papers in training set
Top 2%
1.7%
14
Royal Society Open Science
● 214 papers in training set
Top 3%
1.7%
15
Clinical Trials
○ 11 papers in training set
Top 0.2%
1.3%
16
Frontiers in Bioinformatics
○ 49 papers in training set
Top 0.9%
1.1%
17
Bioinformatics
◐ 1204 papers in training set
Top 8%
1.1%
18
Computers in Biology and Medicine
○ 128 papers in training set
Top 3%
1.1%
19
Value in Health
○ 11 papers in training set
Top 0.2%
1.0%
20
JMIR Research Protocols
◐ 21 papers in training set
Top 1%
0.8%
21
Medicine
○ 31 papers in training set
Top 2%
0.8%
22
Frontiers in Microbiology
○ 427 papers in training set
Top 8%
0.8%
23
F1000Research
○ 88 papers in training set
Top 4%
0.8%
24
Journal of Public Health
◐ 24 papers in training set
Top 1%
0.8%
25
eLife
● 5828 papers in training set
Top 66%
0.8%
26
Healthcare
○ 17 papers in training set
Top 1%
0.6%
27
BMC Medical Informatics and Decision Making
○ 43 papers in training set
Top 2%
0.6%
28
International Journal of Environmental Research and Public Health
○ 128 papers in training set
Top 6%
0.6%

"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.