Label-free, low-cost AI ranking for systematic review screening
Jo, T.; Lee, D.; Lee, L.
Show abstract
Title and abstract screening is among the most labor-intensive steps of systematic reviews. Jev, a low-cost AI decision model, scores records against eligibility criteria without screening labels. We scored 169,982 records from 128 reviews and, after tuning on development reviews, evaluated 23 held-out reviews once and 31 Cochrane reviews post hoc. Jev outranked the best free ranker in all 23 held-out reviews (mean AUC 0.966 vs 0.831) and came within 0.011 AUC of Claude Opus 5.5 on 1999 records at about 1/700 of its list-price cost (US$0.02/1000 records). Combined with active learning, it raised work saved at 95% recall from 0.659 to 0.806. A fixed score threshold fell below 95% recall in 3 of 28 Cochrane reviews, whereas Jev ranking with statistical stopping (post hoc) reached 95% recall in all 51 reviews, reading 86-88% of records. Label-free ranking with Jev was accurate and inexpensive, but stopping safely required screening decisions.
Matching journals
The top 2 journals account for 50% of the predicted probability mass.
Similar papers in this journal
Similar papers in this journal
- Exploring the potential of Claude 2 for risk of bias assessment: Using a large language model to assess randomized controlled trials with RoB 2 94%
- Machine learning for identifying relevant publications in updates of systematic reviews of diagnostic test studies 94%
- Evaluation of the sensitivity, accuracy and currency of the Cochrane COVID-19 Study Register for supporting rapid evidence synthesis production 93%
Similar papers in this journal
- A modular pipeline for natural language processing-screened human abstraction of a pragmatic trial outcome from electronic health records 93%
- Dynamic methods for ongoing assessment of site-level risk in risk-based monitoring of clinical trials: a scoping review 91%
- Use of estimands in cluster randomised trials: a review 90%
Similar papers in this journal
- Evaluation of SURUS: a Named Entity Recognition System to Extract Knowledge from Interventional Study Records 95%
- Completeness of reporting of clinical prediction models developed using supervised machine learning: A systematic review 91%
- Approaches in Analyzing Predictors of Trial Failure: A Scoping Review and Meta-epidemiological study 91%
Similar papers in this journal
- The incubation period of COVID-19: A rapid systematic review and meta-analysis of observational research 94%
- Agreeability testing of AMSTAR-PF, a tool for quality appraisal of systematic reviews of prognostic factor studies 93%
- Protocol for the development of a tool (INSPECT-SR) to identify problematic randomised controlled trials in systematic reviews of health interventions 93%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.