Assessing clinical acuity in the Emergency Department using the GPT-3.5 Artificial Intelligence Model
Williams, C. Y. K.; Zack, T.; Miao, B. Y.; Sushil, M.; Wang, M.; Butte, A. J.
Show abstract
This paper evaluates the performance of the Chat Generative Pre-trained Transformer (ChatGPT; GPT-3.5) in accurately identifying higher acuity patients in a real-world clinical context. Using a dataset of 10,000 pairs of patient Emergency Department (ED) visits with varying acuity levels, we demonstrate that GPT-3.5 can successfully determine the patient with higher acuity based on clinical history sections extracted from ED physician notes. The model achieves an accuracy of 84% and an F1 score of 0.83, with improved performance for more disparate acuity scores. Among the 500 pair subsample that was also manually classified by a resident physician, GPT-3.5 achieved similar performance (Accuracy = 0.84; F1 score = 0.85) compared to the physician (Accuracy = 0.86, F1 score = 0.87). Our results suggest that, in real-world settings, GPT-3.5 can perform comparably to physicians on the clinical reasoning task of ED acuity determination.
Matching journals
The top 3 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- Evaluating large language model workflows in clinical decision support: referral, triage, and diagnosis 95%
- From Tool to Teammate: A Randomized Controlled Trial of Clinician-AI Collaborative Workflows for Diagnosis 93%
- A Framework to Assess Clinical Safety and Hallucination Rates of LLMs for Medical Text Summarisation 93%
Similar papers in this journal
- Automated stratification of trauma injury severity across multiple body regions using multi-modal, multi-class machine learning models 93%
- Empowering Personalized Pharmacogenomics with Generative AI Solutions 93%
- Validation of a Derived International Patient Severity Algorithm to Support COVID-19 Analytics from Electronic Health Record Data 93%
Similar papers in this journal
- Modeling physician variability to prioritize relevant medical record information 93%
- Natural Language Processing for Automated Annotation of Medication Mentions in Primary Care Visit Conversations 92%
- Development and Evaluation of Machine Learning Models for the Detection of Emergency Department Patients with Opioid Misuse from Clinical Notes 92%
Similar papers in this journal
- A Deep Learning Method to Detect Opioid Prescription and Opioid Use Disorder from Electronic Health Records 92%
- Image and structured data analysis for prognostication of health outcomes in patients presenting to the Emergency Department during the COVID-19 pandemic 91%
- Predicting Prognosis in COVID-19 Patients using Machine Learning and Readily Available Clinical Data 91%
Similar papers in this journal
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.