Patient Versus Prediction-Level Evaluation of a Dynamic Clinical Prediction Model of Sepsis
Tuttle, M.; Maas, C. C. H. M.; An, J.; Wessler, B. S.; Harvey, W. F.; Selker, H. P.; van Klaveren, D.; Kent, D. M.
Show abstract
The Epic Sepsis Model version 2 (ESMv2) is a prediction model embedded into the electronic medical record used to warn clinicians which hospitalized patients are at risk for sepsis. We conducted a retrospective cohort study of 31,951 hospitalizations of 25,760 patients to compare analyses conducted at the commonly used patient-level (where a maximum prediction prior to the onset of sepsis is used to measure performance) vs novel prediction-level (where each prediction is used to measure performance). Sepsis, defined by the Sepsis 3 criteria occurred during 1,049 hospitalizations (3.3%). Patient-level analyses suggested excellent discrimination AUC 0.86; [IQR 0.85, 0.87], whereas prediction-level analyses demonstrated lower performance AUC 0.62; [IQR 0.57, 0.65]. Low estimates of the positive predictive value (14.5% at the patient level vs 4% at the prediction level) imply a high number of false alerts. Common evaluation approaches may overstate the performance of dynamic prediction models and mislead clinical decision-making.
Matching journals
The top 6 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- A comparison of machine learning models versus clinical evaluation for mortality prediction in patients with sepsis 96%
- A Sepsis Treatment Algorithm to Improve Early Antibiotic De-escalation While Maintaining Adequacy of Coverage (Early-IDEAS): A Prospective Observational Study 95%
- Development and validation of a cellular host response test as an early diagnostic for sepsis 94%
Similar papers in this journal
- Accuracy of a Risk Alert Threshold for ICU Hypoglycemia: Retrospective Analysis of Alert Performance and Association with Clinical Deterioration Events 93%
- COVID-19 as cause of viral sepsis: A Systematic Review and Meta-Analysis 91%
- An international factorial vignette-based survey of intubation decisions in acute hypoxemic respiratory failure 90%
Similar papers in this journal
- Predictability and Stability Testing to Assess Clinical Decision Instrument Performance for Children After Blunt Torso Trauma 93%
- Identification of physiological adverse events using continuous vital signs monitoring during paediatric critical care transport: a novel data-driven approach 92%
- Generalizability Challenges of Mortality Risk Prediction Models: A Retrospective Analysis on a Multi-center Database 92%
Similar papers in this journal
- Mid-Regional pro-Adrenomedullin (MR-proADM), C-Reactive Protein (CRP) and Other Biomarkers in the Early Identification of Disease Progression in COVID-19 Patients in the Acute NHS Setting 92%
- Extended laboratory panel testing in the Emergency Department for risk-stratification of patients with COVID-19: a single centre retrospective service evaluation 91%
- Kinetics of SARS-CoV-2 antibody responses pre- and post- COVID-19 convalescent plasma transfusion in patients with severe respiratory failure: an observational case-control study 86%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.