Amount and certainty of evidence in Cochrane systematic reviews of interventions: a large-scale meta-research study

ObjectivesTo quantify the amount and certainty of evidence in Cochrane systematic reviews of interventions, and to describe how this evidence has evolved over time. DesignLarge-scale meta-research study Data sourceCochrane Database of Systematic Reviews (search date April 8, 2025) Eligibility criteriaCochrane systematic reviews assessing interventions reporting "Summary of findings" tables. Data extractionData were automatically extracted using web scraping and a large language model, with quality control performed by humans on a random sample. AnalysisWe describe the certainty of evidence for each population-intervention-comparison-outcome (PICO) question reported in all Cochrane "Summary of findings" tables. When available, we compared the certainty of evidence between the initial version and the latest update. ResultsWe identified 5,116 reviews that reported a "Summary of findings" table, containing 64,849 PICO questions. Overall, 24% (n = 15,768) of PICOs had no study included, 31% (n = 20,390) included only 1 study, 14% (n = 8,796) 2 studies, and 31% (n = 19,895) more than 2 studies. Nearly all PICOs (97%) only included randomized trials. The median [Q1-Q3] number of included participants was 123 [0-557]. The certainty of evidence was rated as high for 4% (n = 2,852), moderate for 16% (n = 10,574), low for 27% (n = 17,409), very low for 26% (n = 17,012), and not assessed for 26% (n = 17,002). Of the 7,461 PICO questions with an update (median time to update of 4.3 years [Q1-Q3: 2.6-6.4]), the number of included studies in the latest update remained the same for 63%; the certainty of evidence was unchanged for 71%; upgraded for 13% and downgraded for 15%. ConclusionThe amount and certainty of evidence is low and has not improved over time with review updates. These results question the efficiency of the research ecosystem. SummaryO_ST_ABSWhat is already known on this topicC_ST_ABSO_LIHigh quality up-to-date evidence synthesis is essential for decision-makers. C_LIO_LIConfidence in the evidence informing decision-making can be limited by the amount and quality of primary research on a specific research question C_LI What this study addsO_LIThis large-scale meta-research study analyzed all Cochrane "Summary of findings" tables (i.e., 64,849 population-intervention-comparison-outcome - PICO - questions), and found that about two thirds of the PICO questions were informed by two or fewer studies, with a median [Q1-Q3] of 123 [0-557] participants per PICO; the associated certainty of evidence was rated as high in only 4% of the cases. C_LIO_LIAfter an update of the review (i.e., 7,461 PICOs), 63% PICOs did not include additional studies, and 71% showed no change in certainty of evidence; upgrades and downgrades of certainty occurred at similar frequencies. C_LIO_LIThese results question the efficiency of the research ecosystem. C_LI