Considerations for implementing a machine learning model for suicide risk in the health systems that serve American Indian communities

Objective: We explored the implementation considerations of machine learning (ML) suicide risk identification in a health system that serves primarily American Indian patients in the Southwestern United States. Previous work developed and reported the performance of suicide risk models, but traditional performance metrics do not directly reflect clinical workflows. Materials and Methods: Using EHR data collected for Emergency Department (ED) visits between 10/01/2019 and 10/02/2021, we conducted Decision Curve Analysis (DCA) to identify the standardized net benefit (SNB) of all models previously evaluated and plotted the potential workload implications of the model with the greatest SNB. Finally, we evaluated suicide screening practices during the study period to inform implementation barriers. Results: Among 9,244 patients, 139 (1.5%) had a suicide attempt or death within 90 days of an included visit. The population-specific logistic regression model had the greatest SNB compared to other approaches. When operationalized, the 95th percentile cutoff did not substantially increase clinical workloads, while the 75th percentile cutoff substantially impacted workloads. The secondary analysis on suicide screening depicted variations in screening administration over time. Discussion: While DCA identified advantages of ML suicide risk identification over current standards, the complexities of suicide-related care limited its clinical interpretability. A stepped-care framework may mitigate potential workload burdens. Training initiatives and dynamic implementation strategies will support high-quality implementation of ML suicide risk identification. Conclusion: Our analyses support the feasibility and utility of augmenting existing suicide screening practices with ML suicide risk identification, and demonstrate the importance of nuanced analyses to inform implementation considerations.