Uptake and Implementation of Multiverse-style Analyses Across 613 Studies
Nepomuceno, A.; Ghosal, A.; Sandoval Lentisco, A.; Ioannidis, J. P. A.
Show abstract
Empirical conclusions can depend on the many individual choices researchers make when analyzing data. Multiverse-style analyses address this by computing results across a set of specifications rather than a single one, but it is unclear how widely they are being used and how they are being implemented. We searched the Web of Science Core Collection (May 2026) for articles citing six foundational papers on different variants of multiverse-style methods and classified each of the citing articles as implementing such methods or only discussing them. For implementations, we recorded the framework used, the number of specifications, which of four decision nodes (measurement, data processing, modeling, and estimation) were varied, and other aspects such as how results were visualized and interpreted. Of the 1545 classifiable articles, 613 (39.7%) implemented a multiverse-style analysis (primarily multiverse n = 336, specification curve n = 175, vibration of effects n = 20, multimodel n = 59, and multi-analyst/many-analyst n = 23). Uptake spanned many disciplines, most often psychology (39.8%), the social sciences (20.7%), and medicine (19.2%). The number of specifications ranged from fewer than ten to more than ten thousand (median = 144, IQR 24-1248). Modeling (75%) and data-processing (62%) choices were included most often. Interpretation was predominantly descriptive, whereas formal inference (10.5%), preregistration (8.6%), and explicit attention to the defensibility of specifications (3.9%) were uncommon. Multiverse-style analyses are increasingly becoming established across the quantitative sciences, but some implementation practices can be strengthened so that these analyses become more fully transparent and more genuinely informative about the robustness of research findings.
Matching journals
The top 3 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- Fast and frugal decision tree for the rapid critical appraisal of systematic reviews 95%
- Evaluation of statistical methods used to meta-analyse results from interrupted time series studies: a simulation study 94%
- Development of a search filter to retrieve reports of interrupted time series studies from MEDLINE and PubMed 93%
Similar papers in this journal
- Quantitative bias analysis for mismeasured variables in health research: a review of software tools 93%
- Does advance contact with research participants increase response to questionnaires: A Systematic Review and meta-Analysis 92%
- Completeness of reporting of clinical prediction models developed using supervised machine learning: A systematic review 90%
Similar papers in this journal
- Large language models for conducting systematic reviews: on the rise, but not yet ready for use – a scoping review 93%
- Methods used to select results to include in meta-analyses of nutrition research: a meta-research study 91%
- The use of the Registered Reports format for publication of randomized clinical trials: a cross-sectional study 91%
Similar papers in this journal
- Almost significant: trends and P values in the use of phrases describing marginally significant results in 567,758 randomized controlled trials published between 1990 and 2020 91%
- Assessment of transparency indicators across the biomedical literature: how open is open? 90%
- A comprehensive analysis of the usability and archival stability of omics computational tools and resources 88%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.