How good are large language models for automated data extraction from randomized trials?

In evidence synthesis, data extraction is a crucial procedure, but it is time intensive and prone to human error. The rise of large language models (LLMs) in the field of artificial intelligence (AI) offers a solution to these problems through automation. In this case study, we evaluated the performance of two prominent LLM-based AI tools for use in automated data extraction. Randomized trials from two systematic reviews were used as part of the case study. Prompts related to each data extraction task (e.g., extract event counts of control group) were formulated separately for binary and continuous outcomes. The percentage of correct responses (Pcorr) was tested in 39 randomized controlled trials reporting 10 binary outcomes and 49 randomized controlled trials reporting one continuous outcome. The Pcorr and agreement across three runs for data extracted by two AI tools were compared with well-verified metadata. For the extraction of binary events in the treatment group across 10 outcomes, the Pcorr ranged from 40% to 87% and from 46% to 97% for ChatPDF and for Claude, respectively. For continuous outcomes, the Pcorr ranged from 33% to 39% across six tasks (Claude only). The agreement of the response between the three runs of each task was generally good, with Cohens kappa statistic ranging from 0.78 to 0.96 and from 0.65 to 0.82 for ChatPDF and Claude, respectively. Our results highlight the potential of ChatPDF and Claude for automated data extraction. Whilst promising, the percentage of correct responses is still unsatisfactory and therefore substantial improvements are needed for current AI tools to be adopted in research practice. Highlights1. What is already knownO_LIIn evidence synthesis, data extraction is a crucial procedure, but it is time intensive and prone to human error, with reported data extraction error rates at meta-analyses level reaching up to 67%. C_LIO_LIThe rise of large language models (LLMs) in the field of artificial intelligence (AI) offers a solution to these problems through automation. C_LI 2. What is newO_LIIn this case study, we investigated the performance of two AI tools for data extraction and confirmed that AI tools can reach the same or better performance than humans in terms of data extraction from randomized trials for binary outcomes. C_LIO_LIHowever, AI tools performed poorly at extracting data from continuous outcomes. C_LI 3. Potential impact for Research Synthesis Methods readers outside the authors fieldO_LIOur study suggests LLMs have great potential in assisting data extraction in evidence syntheses through (semi-)automation. Further efforts are needed to improve accuracy, especially for continuous outcomes data. C_LI