Interpretation of wide confidence intervals in meta-analytic estimates: Is the 'Absence of Evidence' 'Evidence of Absence'?

IntroductionRecently, a Cochrane review by Jefferson et al. on physical interventions to slow the spread of respiratory viruses concluded that, "Wearing masks in the community probably makes little or no difference to the outcome of laboratory-confirmed influenza/SARS-CoV-2 compared to not wearing masks", though this finding had a wide confidence interval. Cochrane issued a rare clarifying statement, fueling controversy. We sought to contextualize the findings of the review by Jefferson et al. MethodsWe searched for consecutive reviews by Cochrane published on or before March 9th, 2023. We included studies where a central finding showed an intervention offered no statistically significant benefit, and ascertained the language used by reviewers to describe that result. We compare this to the report by Jefferson et al., and deemed it consistent or inconsistent with the language of their report. ResultsWe found between November 21st, 2022, and March 9th, 2023, there were 20 Cochrane reviews that met the inclusion criteria. We found that 95% (n = 19) of the reviews used language that was consistent with Jeffersons findings, while 5% (n = 1) used language inconsistent with Jeffersons conclusion, describing the effect of the intervention on the outcome as "unclear". DiscussionMost reviews performed by Cochrane conclude that interventions which fail to show statistically significant benefits make "no difference" have "no effect" or do not "increase or decrease" the outcome, and this occurs despite wide confidence intervals. The conclusions by Jefferson et al. are consistent with Cochrane reporting guidelines and clarification from the organization was unjustified.