What we do in the shadows: Methodologists use a range of synthesis methods when meta-analysis of all results is not possible but describe challenges in planning and selecting methods

Introduction Systematic review authors commonly encounter situations where the data required for meta-analysis are incompletely reported (e.g. when effect estimates are reported without a measure of precision). In this circumstance, many systematic review authors use a method other than meta-analysis (e.g. vote counting), but rarely describe those methods or the rationale for selecting them. We aimed to investigate what methods authors consider when meta-analysis of all study results is not possible, and what factors influence their decisions. Methods We interviewed 12 experienced systematic review authors, editors and methodologists, presenting four scenarios in which it was not possible to combine all results using meta-analysis. Scenarios varied in the number and size of included studies, available data, and risk of bias. Participants discussed the methods they considered to summarise, synthesise and present the results; whether they would synthesise available results; and how they would draw overall conclusions. Results Factors that informed decisions included participants' overall purpose in conducting synthesis, existing beliefs about study results and synthesis methods, trust in the available data, and the decision-making needs of end users. Participants differed in which synthesis methods to use, whether they would use multiple synthesis methods, and which studies they would analyse with each method. Conclusions We identified several synthesis methods considered when meta-analysis of all results is not possible, and factors that influence the selection of methods, neither of which are routinely reported. More complete reporting of these methods and the factors informing decisions would allow readers to better understand the decisions made.