Predicting Opportunities for Improvement in Trauma using Machine Learning: A Registry Based Study

ImportanceTrauma quality improvement programs relies on peer review of patient cases to identify opportunities for improvement. Current state-of-the-art systems for selecting patient cases for peer review use audit filters that struggle with poor performance. ObjectiveTo develop models predicting opportunities for improvement in trauma care and compare their performance to currently used audit filters. Design, Setting and ParticipantsThis single-center registry-based cohort study used data from the trauma centre at Karolinska University Hospital in Stockholm, Sweden, between 2013 and 2023. Participants were adult trauma patients included in the local trauma registry. The models predicting opportunities for improvement in trauma care were developed using logistic regression and the eXtreme Gradient Boosting learner (XGBoost) with an add-one-year-in expanding window approach. Performance was measured using the integrated calibration index (ICI), area under the receiver operating curve (AUC), true positive rates (TPR) and false positive rates (FPR). We compared the performance of the models to locally used audit filters. Main outcome measureOpportunities for improvement, defined as preventable events in patient care with adverse outcomes. These opportunities for improvement were identified by the local peer review processes. ResultsA total of 8,220 patients were included. The mean (SD) age was 45 (21), 5696 patients (69%) were male, and the mean (SD) injury severity score was 12 (13). Opportunities for improvement were identified in 496 (6%) patients. The logistic regression and XGBoost models were well calibrated with ICIs (95% CI) of 0.032 (0.032-0.032) and 0.033 (0.032-0.033). Compared to the audit filters, both the logistic regression and XGBoost models had higher AUCs (95% CI) of 0.72 (0.717-0.723) and 0.75 (0.747-0.753), TPR (95% CI) of 0.885 (0.881-0.888) and 0.904 (0.901-0.907), and lower FPR (95% CI) of 0.636 (0.635-0.638) and 0.599 (0.598-0.6). The audit filters had an AUC (95% CI) of 0.616 (0.614-0.618), a TPR (95% CI) of 0.903 (0.9-0.906), and a FPR (95% CI) of 0.671 (0.67-0.672). Conclusion and RelevanceBoth the logistic regression and XGBoost models outperformed audit filters in predicting opportunities for improvement among adult trauma patients and can potentially be used to improve systems for selecting patient cases for trauma peer review. Key pointQuestion: How does the performance of machine learning models compare to audit filters when screening for opportunities for improvement, preventable events in care with adverse outcomes, among adult trauma patients? Findings: Our registry-based cohort study including 8,220 patients showed that machine learning models outperform audit filters, with improved discrimination and false-positive rates. Compared to audit filters, these models can be configurated to balance sensitivity against overall screening burden. Meaning: Machine learning models have the potential to reduce false positives when screening for opportunities for improvement in the care of adult trauma patients and thereby enhancing trauma quality improvement programs.