Large-Language-Model Mortality Risk Stratification in the Intensive Care Unit: A Benchmark Against APACHE II

BackgroundAccurately predicting clinical trajectories in critically ill patients remains challenging due to physiological instability and multisystem organ dysfunction. Traditional prognostic tools, such as the APACHE II score, offer standardized risk assessment but are constrained by static algorithms. This study evaluates the predictive performance and reliability of large language models (LLMs) compared to APACHE II for in-hospital mortality prediction. MethodsThis was a single-center, retrospective study. De-identified clinical data from 70 critically ill patients were provided to four LLMs--Gemini, Llama, GPT-4, and R1. Each model stratified patients into high-, intermediate-, or low-risk (of in-hospital death) categories without being instructed to apply the APACHE II method. To assess the impact of additional information, models were also provided with de-identified hospital discharge summaries from prior hospital admissions. Consistency and rationale analyses were performed across multiple iterations. FindingsLLMs demonstrated a general tendency toward risk overestimation, classifying more patients as high risk compared to APACHE II. Mortality rates within high-risk groups were lower than APACHE-predicted rates, suggesting calibration mismatch. Gemini, when supplemented with additional clinical context, uniquely identified a low-risk group. Gemini, GPT-4, and R1 exhibited the highest consistency across repeated evaluations, while Llama showed greater variability that improved with context. Semantic rationale analyses revealed greater stability among larger models, indicating non-stochastic reasoning patterns. ConclusionsLLMs, supplemented with discharge summaries from prior hospitalizations, show promise in mortality risk stratification in critically ill patients. However, further refinement is necessary to improve calibration and reliability before clinical implementation. Context-aware prompting strategies and improved model calibration may enhance the utility of LLMs alongside established systems like APACHE II. Author SummaryPredicting which critically ill patients are at greatest risk of dying in the hospital is one of the most important and difficult tasks faced by doctors. Traditionally, weve used structured scoring systems like APACHE II, which rely on a fixed set of patient measurements. In this study, we explored whether large language models (LLMs)--the same kind of technology behind chatbots like ChatGPT--could perform this task just as well, or even better. We provided four different LLMs with real patient data from our intensive care unit and asked them to assess each patients risk of dying, without giving them any instructions about how to do so. We also tested whether adding more context, such as hospital discharge summaries, made their predictions more accurate or consistent. We found that while LLMs tended to overestimate risk, some models--especially when given extra clinical information--showed strong consistency and thoughtful reasoning in their predictions. Our findings suggest that LLMs may eventually serve as helpful partners to physicians, offering a flexible and adaptable way to interpret complex clinical data. However, more work is needed to ensure that these tools are safe, reliable, and transparent before they can be used in real-world hospital settings