Evaluating the Portability of SepsisWatch: A Multi-Site External Validation of a Sepsis Machine Learning Model
Valan, B. N.; Prakash, A.; Ratliff, W. N.; Gao, M.; Muthya, S.; Thomas, A.; Eaton, J. L.; Gardner, M.; Nichols, M.; Revoir, M.; Tart, D.; O'Brien, C.; Patel, M.; Balu, S.; Sendak, M.
Show abstract
ImportanceSepsis accounts for a substantial portion of global deaths and healthcare costs. Early detection using machine learning (ML) models offers a critical opportunity to improve care and reduce the burden of sepsis. ObjectiveTo externally validate the SepsisWatch ML model, initially developed at Duke University, in a community healthcare and assess its performance and clinical utility in early sepsis detection. DesignThis retrospective external validation study evaluated the performance of the SepsisWatch model in a new environment. Data from patient encounters at Summa Healths emergency departments between 2020 and 2021 were used. The study analyzed the models ability to predict sepsis using a combination of static and dynamic patient data. SettingThe study was conducted at Summa Health, a nonprofit healthcare system in Northeast Ohio, covering two emergency departments (EDs) associated with acute care hospitals, and two standalone EDs. ParticipantsEncounters associated with adult patients in any of Summa Healths four EDs were included. Encounters lasting <1 hour were excluded. Only the first 36 hours of each encounter were used in model evaluation. Intervention(s)/Exposure(s)The SepsisWatch model was used to predict sepsis based on patient data. Main Outcome(s) and Measure(s)The primary outcomes measured were the models area under the precision-recall curve (AUPRC), and area under the receiver operator curve (AUROC). ResultsThe study included 205,005 encounters from 101,584 unique patients. 54.7% (n = 112,223) patients were female and the mean age was 50 (IQR, [38,71]). The model demonstrated strong performance across the Summa Health system, with little variation across different sites. The AUROC ranged from 0.906 to 0.960, and the AUPRC ranged from 0.177 to 0.252 across the four sites. Conclusions and RelevanceThe external validation of the SepsisWatch model in a community health system setting confirmed its robust performance and portability across different geographical and demographic contexts. The study underscores the potential of advanced ML models in improving sepsis detection in both academic and community hospital settings, paving the way for prospective studies to measure the clinical and operational impact of such models in healthcare.
Matching journals
The top 5 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- A comparison of machine learning models versus clinical evaluation for mortality prediction in patients with sepsis 96%
- Clinical prediction rule for SARS-CoV-2 infection from 116 U.S. emergency departments 95%
- Derivation and validation of a triage tool for acutely ill adults with suspected COVID-19: The PRIEST observational cohort study 95%
Similar papers in this journal
- A deep learning model for clinical outcome prediction using longitudinal inpatient electronic health records 94%
- A Machine Learning Approach to Identifying Delirium from Electronic Health Records 92%
- Enhancing Research Data Infrastructure to Address the Opioid Epidemic: The Opioid Overdose Network (02-Net) 92%
Similar papers in this journal
- Predictability and Stability Testing to Assess Clinical Decision Instrument Performance for Children After Blunt Torso Trauma 94%
- Identification of physiological adverse events using continuous vital signs monitoring during paediatric critical care transport: a novel data-driven approach 93%
- Identification of predictive patient characteristics for assessing the probability of COVID-19 in-hospital mortality 93%
Similar papers in this journal
- Score for Emergency Risk Prediction (SERP): An Interpretable Machine Learning AutoScore–Derived Triage Tool for Predicting Mortality after Emergency Admissions 96%
- Consistency of performance of adverse outcome prediction models for hospitalized COVID-19 patients 93%
- Prospective and External Evaluation of a Machine Learning Model to Predict In-Hospital Mortality 91%
Similar papers in this journal
- Predicting bloodstream infection outcome using machine learning 94%
- AKI Risk Score (AKI-RiSc): Developing an Interpretable Clinical Score for Early Identification of Acute Kidney Injury for Patients Presenting to the Emergency Department 94%
- Using explainable machine learning to identify patients at risk of reattendance at discharge from emergency departments 93%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.