Comparison of Multimodal Deep Learning Approaches for Predicting Clinical Deterioration in Ward Patients

ObjectiveImplementing machine learning models to identify clinical deterioration on the wards is associated with improved outcomes. However, these models have high false positive rates and only use structured data. Therefore, we aim to compare models with and without information from clinical notes for predicting deterioration. Materials and MethodsAdults admitted to the wards at the University of Chicago (development cohort) and University of Wisconsin-Madison (external validation cohort) were included. Predictors consisted of structured and unstructured variables extracted from notes as Concept Unique Identifiers (CUIs). We parameterized CUIs in five ways: Standard Tokenization (ST), ICD Rollup using Tokenization (ICDR-T), ICD Rollup using Binary Variables (ICDR-BV), CUIs as SapBERT Embeddings (SE), and CUI Clustering using SapBERT embeddings (CC). Each parameterization method combined with structured data and structured data-only were compared for predicting intensive care unit transfer or death in the next 24 hours using deep recurrent neural networks. ResultsThe study included 506,076 ward patients, 4.9% of whom experienced the outcome. The SE model achieved the highest AUPRC (0.208), followed by CC (0.199) and the structured-only model (0.199), ICDR-BV (0.194), ICDR-T (0.166), and ST (0.158). The CC and structured-only models achieved the highest AUROC (0.870), followed by ICDR-T (0.867), ICDR-BV (0.866), ST (0.860), and SE (0.859). DiscussionA multimodal model combining structured data with embeddings using SapBERT had the highest AUPRC, but performance was similar between models with and without CUIs. ConclusionThe addition of CUIs from notes to structured data did not meaningfully improve model performance for predicting clinical deterioration. Lay SummaryImplementing machine learning models to identify clinical deterioration on the wards is associated with improved outcomes. However, these models have high rates of false positives and only use structured electronic health record (EHR) data as predictor variables. Therefore, we aimed to determine if information from clinical notes improves performance and compare methods of combining information from clinical notes with structured data. Model features consisted of variables from structured EHR data (e.g., vital signs, laboratory results) and Concept Unique Identifiers (CUIs) extracted from notes. We parameterized the CUIs in five ways, each of which was combined with the structured data to predict intensive care unit transfer or death in the next 24 hours with a multimodal deep recurrent neural network. The study cohort included 506,076 ward patients across two health systems, 4.9% of whom experienced the outcome. During external validation, the models using both CUIs and structured data performed similarly to the model using only structured data. Of the CUI parameterization approaches, methods that used SapBERT embeddings had the highest discrimination. Our findings show that, while adding medical concept variables from clinical notes could provide additional clinical context for clinicians, they do not enhance model performance compared to structured data alone.