Comparing computable structured phenotype- versus large language model-identification of opioid use disorder using electronic health record data
Molina, M. F.; Fenton, C.; LeSaint, K. T.; Pimentel, S. D.; Kornblith, A. E.
Show abstract
Study ObjectiveTo compare a computable structured opioid use disorder (OUD) phenotype currently used to trigger emergency department (ED) clinical decision support (CDS) with a large language model (LLM) for OUD identification, using expert physician review as the reference standard. MethodsWe conducted a retrospective study of randomly sampled adult ED encounters (January 1, 2023-October 17, 2024) at a single academic health system. Encounters were stratified by structured phenotype status and weighted to reflect population prevalence. The structured phenotype, implemented operationally to activate CDS, incorporated diagnosis codes, medications for OUD, urine toxicology results, addiction consultations, and keyword recognition. An LLM (ChatGPT 4.1) analyzed ED notes from the index visit using a zero-shot prompt. Two board-certified emergency physicians independently determined OUD status by full chart review; discrepancies were adjudicated by a third reviewer. We calculated weighted sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). ResultsAmong 302 encounters, weighted OUD prevalence was 5.6% (95% CI 4.0-7.0%). The structured phenotype demonstrated sensitivity 0.84 (95% CI 0.42-0.97) and specificity 0.964 (95% CI 0.96-0.97) (PPV 0.58; NPV 0.99). The LLM demonstrated sensitivity 0.81 (95% CI 0.70-0.88) and specificity 0.996 (95% CI 0.993-0.998) (PPV 0.92; NPV 0.99). Specificity was significantly higher for the LLM (p<0.0001). ConclusionBoth approaches demonstrated strong diagnostic performance. Although the structured phenotype showed slightly higher sensitivity, the LLM achieved higher specificity and PPV, suggesting potential to reduce false-positive alerts in ED workflows. Prospective validation in larger, representative populations is needed to guide clinical implementation.
Matching journals
The top 4 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- Enhancing Research Data Infrastructure to Address the Opioid Epidemic: The Opioid Overdose Network (02-Net) 96%
- Development and Evaluation of Machine Learning Models for the Detection of Emergency Department Patients with Opioid Misuse from Clinical Notes 94%
- Characterization and Racial Stratification of Social Determinants of Health for Individuals with Type 2 Diabetes as Recorded in Electronic Health Records: Implications for Artificial Intelligence Development 92%
Similar papers in this journal
- Score for Emergency Risk Prediction (SERP): An Interpretable Machine Learning AutoScore–Derived Triage Tool for Predicting Mortality after Emergency Admissions 93%
- Electronic Health Record Documentation of Psychiatric Assessments in Massachusetts Emergency Department and Outpatient Settings During the COVID-19 Pandemic 92%
- Low adherence to existing model reporting guidelines by commonly used clinical prediction models 91%
Similar papers in this journal
- Two Data-Driven Approaches to Identifying the Spectrum of Problematic Opioid Use: A Pilot Study within a Chronic Pain Cohort 94%
- A Deep Learning Method to Detect Opioid Prescription and Opioid Use Disorder from Electronic Health Records 92%
- Analyzing Pain Patterns in the Emergency Department: Leveraging Clinical Text Deep Learning Models for Real-World Insights 92%
Similar papers in this journal
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.