Effects of non-pharmaceutical interventions on COVID-19: A Tale of Two Models

In this paper, we compare the inference regarding the effectiveness of the various non-pharmaceutical interventions (NPIs) for COVID-19 obtained from three SIR models, all developed by the Imperial College COVID-19 Response Team. One model was applied to European countries and published in Nature1 (model 1), concluding that complete lockdown was by far the most effective measure, responsible for 80% of the reduction in Rt, and 3 million deaths were avoided in the examined countries. The Imperial College team applied a different model to the USA states2 (model 2), and in response to our original submission, the Imperial team has proposed in a referee report a third model which is a hybrid of the first two models (model 3). We demonstrate that inference is highly nonrobust to model specification. In particular, inference regarding the relative effectiveness of NPIs changes substantially with the model and decision makers who are unaware of, or ignore, model uncertainty are underestimating the risk attached to any decisions based on that model. Our primary observation is that by applying to European countries the model that the Imperial College team used for the USA states (model 2), complete lockdown has no or little effect, since it was introduced typically at a point when Rt was already very low. Moreover, using several state-of-the-art metrics for Bayesian model comparison, we demonstrate that model 2 (when applied to the European data) is better supported by the data than the model published in Nature1. In particular, serious doubt is cast on the conclusions in Flaxman et al.1, whether we examine the data up to May 5th (as in Flaxman et al.1) or beyond the point when NPIs began to be lifted. Only by objectively considering a wide variety of models in a statistically principled manner, can one begin to address the effectiveness of NPIs such as lockdown. The approach outlined in this paper provides one such path.