Back

Usability and Accuracy of the SWIFT-ActiveScreener Preliminary evaluation for use in clinical research

Liu, J. W.; Ein, N.; Gervasio, J.; Easterbrook, B.; Nouri, M. S.; Nazarov, A.; Richardson, J. D.

2023-08-25 psychiatry and clinical psychology
10.1101/2023.08.24.23294573 medRxiv
Show abstract

Systematic reviews (SRs) employ standardized methodological processes for synthesizing empirical evidence to answer specific research questions. These processes include rigorous screening phases to determine eligibility of articles against strict inclusion and exclusion criteria. Despite these processes, SRs are a significant undertaking, and this type of research often necessitates extensive human resource requirements, especially when the scope of the review is large. Given the substantial resources and time commitment required, we investigated a way in which the screening process might be accelerated while maintaining high fidelity and adherence to SR processes. More recently, researchers have increasingly turned to artificial intelligence-based (AI) software to expedite the screening process. This paper evaluated the accuracy and usabiity of a novel, machine learning program, Sciome SWIFT-ActiveScreener (ActiveScreener) in a large SR of mental health outcomes following treatment for PTSD. ActiveScreener exceeded the expected 95% accuracy of the program to predict inclusion or exclusion of relevant articles, and was reported to be user friendly by both novice and seasoned screeners. Our results showed that ActiveScreener, when used appropriately, may save considerable time and human resources when performing SR.

Matching journals

The top 8 journals account for 50% of the predicted probability mass.

1
PLOS ONE
● 5266 papers in training set
Top 12%
15.4%
Similar papers in this journal
2
Journal of Medical Internet Research
◐ 87 papers in training set
Top 0.2%
9.1%
Similar papers in this journal
3
Systematic Reviews
○ 15 papers in training set
Top 0.1%
5.7%
Similar papers in this journal
4
PLOS Digital Health
● 106 papers in training set
Top 1%
4.9%
Similar papers in this journal
5
JMIR Research Protocols
◐ 21 papers in training set
Top 0.1%
4.9%
Similar papers in this journal
6
BMJ Open
● 601 papers in training set
Top 5%
4.1%
7
Research Synthesis Methods
○ 20 papers in training set
Top 0.1%
3.3%
8
DIGITAL HEALTH
○ 17 papers in training set
Top 0.2%
3.2%
50% of probability mass above
9
Frontiers in Psychiatry
○ 87 papers in training set
Top 1.0%
2.2%
10
JMIR Medical Informatics
◐ 18 papers in training set
Top 0.3%
2.2%
11
BMC Medical Informatics and Decision Making
○ 43 papers in training set
Top 0.9%
1.9%
12
Scientific Reports
○ 3612 papers in training set
Top 51%
1.8%
13
JMIR mHealth and uHealth
◐ 11 papers in training set
Top 0.1%
1.8%
14
BMC Health Services Research
○ 51 papers in training set
Top 1%
1.8%
15
JMIR Formative Research
◐ 33 papers in training set
Top 0.8%
1.8%
16
Nature Protocols
○ 33 papers in training set
Top 0.3%
1.5%
17
Frontiers in Artificial Intelligence
○ 20 papers in training set
Top 0.3%
1.5%
18
JAMIA Open
◐ 42 papers in training set
Top 1%
1.5%
19
Contemporary Clinical Trials Communications
○ 11 papers in training set
Top 0.3%
1.1%
20
Journal of Clinical Epidemiology
○ 31 papers in training set
Top 0.6%
1.1%
21
Frontiers in Public Health
○ 148 papers in training set
Top 5%
1.1%
22
PeerJ
◐ 308 papers in training set
Top 7%
1.1%
23
F1000Research
○ 88 papers in training set
Top 2%
1.1%
24
Medicine
○ 31 papers in training set
Top 1%
1.1%
25
Frontiers in Digital Health
○ 24 papers in training set
Top 0.8%
1.1%
26
Journal of Public Health
◐ 24 papers in training set
Top 0.8%
1.1%
27
BMC Biology
○ 265 papers in training set
Top 4%
1.1%
28
Journal of Clinical and Translational Science
◐ 14 papers in training set
Top 0.4%
1.1%
29
IEEE Access
● 35 papers in training set
Top 1%
0.9%
30
Frontiers in Psychology
○ 56 papers in training set
Top 1%
0.9%

"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.