Assessment of Bias in Clinical Trials with LLMs Using ROBUST-RCT: A Feasibility Study.

BACKGROUNDBias assessment is a crucial step in evaluating evidence from randomized controlled trials. The widely adopted Cochrane RoB 2, designed to identify these issues, is complex, resource-intensive, and unreliable. Advances in artificial intelligence (AI), particularly in the field of large language models (LLMs), now allow the automation of complex tasks. While prior investigations have focused on whether LLMs could perform assessments with RoB 2, integrating technologies does not resolve the intrinsic methodological issues of the instrument. This is the first feasibility study to evaluate the reliability of ROBUST-RCT, a novel bias assessment tool, as applied by humans and LLMs. METHODSA sample of RCTs of drug interventions was screened for eligibility. Reviewers working independently used ROBUST-RCT to assess different aspects of the studies and then reached a consensus through discussion. A chain-of-thought prompt instructed four LLMs on how to apply ROBUST-RCT. The primary analysis used Gwets AC2 coefficient and benchmarking to assess inter-rater reliability of the "judgment set", defined as the series of final assessments for the six core items in the ROBUST-RCT tool. RESULTS54 assessments of each LLM were compared to human consensus in the primary analysis. Gwets AC2 inter-rater reliability ranged from 0.46 to 0.69. With 95% confidence, three of the four tested LLMs achieved moderate or higher reliability based on probabilistic benchmarking. A secondary analysis also found a Fleiss Kappa of 0.49 (95% CI: 0.30 - 0.60) between human reviewers before consensus, numerically higher than the values reported in prior literature about RoB 2. CONCLUSIONLarge Language Models (LLMs) can effectively perform risk-of-bias assessments using the ROBUST-RCT tool, enabling their integration into future systematic review workflows aiming for enhanced objectivity and efficiency.