External validation, recalibration and updating of the OxSATS risk model for suicide after self-harm in England

BackgroundExternal validations of existing risk models is an efficient step towards potential implementation, obviating the need to develop new models. However, validation in new clinical settings poses several challenges. ObjectiveTo externally validate the OxSATS tool using data from the Oxford Monitoring System for Self-harm in England. OxSATS is a validated tool to predict suicide after self-harm developed using Swedish population registers. MethodsWe selected episodes of self-harm (ICD-10 codes X60-84; Y10-34) by individuals aged 10-64 years who presented to a large regional hospital between 1 January 2000 and 31 December 2018, and were followed up until 31 December 2019. We applied the OxSATS tool to estimate each individuals suicide risk within 12 months after their index self-harm. We assessed model performance using discrimination (Harrells c-index) and calibration measures (calibration plot and the observed-to-expected events ratio, O:E). We assessed the effects of missing predictors on calibration and subsequently recalibrated the model. FindingsWe identified 16,120 individuals who presented to hospital with self-harm, of whom 101 (0.6%) died by suicide in the 12-month follow-up period. The OxSATS model showed good discrimination in external validation (c-index=0.72, 95% CI=0.67, 0.77). Recalibration was required because initial calibration reflected a lower outcome rate in the new data. After recalibration, calibration performance was excellent (O:E=1.00, 95% CI=0.80, 1.20). ConclusionsDespite differences in clinical services and outcome ascertainment, suicide risk models can maintain good predictive performance in new settings. However, recalibration should be considered when applying prediction models in new settings, and the impact of missing predictors should be assessed using sensitivity analyses. KEY MESSAGESO_ST_ABSWhat is already known on this topicC_ST_ABSSuicide risk is substantially elevated after hospital presentation for self-harm, but most existing risk assessment tools rely on rating scales or binary cut-offs, show limited predictive accuracy, and rarely report calibration. OxSATS is a prognostic model developed using Swedish register data that provides continuous risk estimates and demonstrated good discrimination and calibration in its original setting. External validation in new healthcare systems is essential before implementation, but is often complicated by differences in predictor definitions, missing variables, and outcome prevalence. What this study addsThis study provides the first external validation of OxSATS in an English clinical setting using routinely collected hospital data. The model retained good discrimination but initially overpredicted suicide risk due to a lower baseline event rate and one missing predictor, highlighting the importance of calibration assessment. How this study might affect research, practice or policyFuture research and implementation strategies should routinely incorporate external validation, sensitivity analyses for missing predictors, and local recalibration before clinical or policy adoption.