Detecting simulated pathogen releases in a real-world health data set

The purpose of electronic disease syndromic surveillance (EDSyS) systems is to detect hazardous pathogens and other unusual signals in health surveillance data before such events are identified by an individual clinician or healthcare facility. However, EDSyS systems have primarily been evaluated using simulated health surveillance data, which do not necessarily capture the richness and complexities of real-world health data. We have updated and extended an existing EDSyS system, EpiDefend, which combines ensemble forecasting and recursive Bayesian estimation in a particle filter framework that supports demographic and spatial structure. We simulated the release of several pathogens, both infectious and non-infectious, and injected the resulting cases into a real-world health data set. Here we evaluate EpiDefend's sensitivity and specificity in detecting these simulated releases, and measure the time to detection against pathogen-specific estimates of the time to clinical detection, as informed by clinicians and microbiologists. We show that for diseases where clinical diagnosis can be challenging, such as Q fever (Coxiella burnetii) and tularaemia (Francisella tularensis), EpiDefend can reliably beat the time to clinical detection. In contrast, for pathogens that can be clinically diagnosed relatively quickly, such as inhalational anthrax and pneumonic plague, it is extremely difficult to beat the time to clinical detection. Our results suggest that EpiDefend may be able to reliably detect real-world introductions or releases of some pathogens at low false-alarm rates before a clinical diagnosis would be confirmed, and this would represent a landmark achievement for EDSyS systems.