Comparative performance of the concurrent comparator design with existing vaccine safety surveillance approaches on real-world observational health data

BackgroundIt is critical public health concern to identify safety signals originating from wide-scale immunization efforts. Such safety signals may be identified from spontaneous reports and other data sources. Although some work has been done on the best methods for vaccine safety surveillance, there is a scarcity of information on how these perform in analyses of real-world data. MethodsWe use four administrative claims databases and one electronic health record (EHR) database to evaluate the operating characteristics of the recently proposed concurrent comparator, self-controlled case series, historical comparator and case-control epidemiological designs for vaccine safety, using negative control outcomes (unrelated to the vaccine), imputed positive control outcomes, and one real-world positive control outcome (myocarditis or pericarditis) for COVID-19. In this evaluation, we consider vaccine exposures for COVID-19, 2017-2018 seasonal influenza, H1N1pdm flu, Human Papillomavirus (HPV), and Varicella-Zoster. The methods are compared based on type 1 error, power of association detection, and proportion of non-finite association estimates produced. ResultsAll methods exhibit systematic error, leading to type 1 errors that are greater than the nominal (= 0.05) threshold, often by a substantial amount. To restore near-nominal type 1 error, we carry out empirical calibration based on the large set of negative controls. Post-empirical calibration, the self-controlled case series designs had the highest power overall, closely followed by the concurrent comparator designs. However, concurrent comparator analyses often produced a higher proportion of non-finite estimates. ConclusionOur results indicate that there remains non-negligible systematic error under the concurrent comparator. In terms of statistical performance, the concurrent comparator designs show promising results in some scenarios, regularly outperforming the historical comparator and case-control designs, but often producing non-finite estimates. Future work building on the concurrent comparator design is required to construct more efficient designs with lower systematic error.