Balanced Risk Integrity Index: a stratum-level metric for evaluating clinical prediction alerts under competing risks and censoring
Jeong, I.; Lee, T.; Park, J.-H.; Kim, Y.; Lee, H.
Show abstract
Background: Conventional discrimination metrics, such as the area under the receiver operating characteristic curve (AUROC), do not characterize the composition of outcomes among patients flagged as high risk by clinical prediction models. This limitation is particularly relevant in critical care, where competing deaths and incomplete follow-up can complicate the interpretation of prediction alerts. We developed the Balanced Risk Integrity Index (BRII), a stratum-level metric defined as the proportion of target events among target events, competing deaths, and early-censored observations in a flagged high-risk stratum. BRII is algebraically related to positive predictive value (PPV) and is reported alongside its component decomposition to characterize the outcome composition of flagged patients. Methods: We evaluated BRII using seven acute kidney injury (AKI) prediction models developed in MIMIC-IV and externally validated in MIMIC-III and eICU-CRD (116,010 patients in total). Additional simulations examined the effects of competing deaths and early censoring on BRII and conventional model performance metrics. Results: For 48-hour AKI prediction in the top 10% risk stratum, BRII ranged from 0.90 to 0.96 in MIMIC-III, compared with less than 0.26 across all models in eICU-CRD, where 61.5% of evaluated patients were censored before the prediction horizon. In the eICU-CRD top 10% strata for the same target, 37.0% to 50.2% of patients were censored early, 12.2% to 15.5% had AKI, and 3.4% to 8.1% died before AKI. BRII and PPV ordered at least one model pair differently in 34 of 48 cohort-outcome-top-k groups. Under simulated random censoring for 48-hour AKI prediction, BRII declined by up to 0.80 while binary-label AUROC changed by less than 0.02. Conclusions: BRII provided complementary information on high-risk stratum composition, yielding different model rankings from PPV and revealing substantial differences across external validation cohorts. Interpreted alongside PPV and its component decomposition, BRII offers a standardized approach to evaluating clinical prediction alerts across models and observation environments.
Matching journals
The top 7 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- AKI Risk Score (AKI-RiSc): Developing an Interpretable Clinical Score for Early Identification of Acute Kidney Injury for Patients Presenting to the Emergency Department 94%
- Machine learning for classifying chronic kidney disease and predicting creatinine levels using at-home measurements 92%
- Predicting bloodstream infection outcome using machine learning 92%
Similar papers in this journal
- Urinary tract infections in children: building a causal model-based decision support tool for diagnosis with domain knowledge and prospective data 91%
- Towards reduction in bias in epidemic curves due to outcome misclassification through Bayesian analysis of time-series of laboratory test results: Case study of COVID-19 in Alberta, Canada and Philadelphia, USA 89%
- Completeness of reporting of clinical prediction models developed using supervised machine learning: A systematic review 89%
Similar papers in this journal
- Development and Validation of ‘Patient Optimizer’ (POP) Algorithms for Predicting Surgical Risk with Machine Learning 93%
- OASIS+: leveraging machine learning to improve the prognostic accuracy of OASIS severity score for predicting in-hospital mortality 91%
- Optimized Feature Selection and Advanced Machine Learning for Stroke Risk Prediction in Revascularized Coronary Artery Disease Patients 91%
Similar papers in this journal
- Causal modeling of chronic kidney disease in a participatory framework for informing the inclusion of social drivers in health algorithms 94%
- Personalizing renal replacement therapy initiation in the intensive care unit: a reinforcement learning-based strategy with external validation on the AKIKI randomized controlled trials 92%
- Real-Time Electronic Health Record Mortality Prediction During the COVID-19 Pandemic: A Prospective Cohort Study 91%
Similar papers in this journal
- Evaluating the kidney disease progression using a comprehensive patient profiling algorithm: A hybrid clustering approach 93%
- The Chronic Kidney Disease and Acute Kidney Injury Involvement in COVID-19 Pandemic: A Systematic Review and Meta-analysis 92%
- A comparison of machine learning models versus clinical evaluation for mortality prediction in patients with sepsis 92%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.