Characterizing Documented Psychosocial Stressors in Pediatric Psychiatric Emergencies with an Open-Weight Large Language Model
Hartlage, C. S.; Manning, E. R.; Bernard, J.; Vaish, S.; Gray, J.; Young, M.; Pestian, T.; Folger, A. T.; Tachinardi, P.; Mendonca, E. A.; Brokamp, C.
Show abstract
Objective: To evaluate whether a locally hosted open-weight large language model (LLM) can extract documented psychosocial factors from pediatric psychiatric intake notes and apply validated extraction to a large emergency psychiatry cohort. Materials and Methods: We identified emergency department presentations at Cincinnati Children's Hospital Medical Center from January 1, 2016, through December 31, 2024, among patients younger than 18 years with psychiatric billing diagnoses. Using full-text intake notes, gpt-oss:120b classified peer conflict, sleep disruption, and school-related academic, attendance, and disciplinary issues as detected, negated, or indeterminate. Four human raters independently reviewed 50 notes. We compared Fleiss' kappa among humans alone versus humans plus the LLM, assessed repeated-query stability across 50 independent calls per note, and applied the workflow to all eligible notes. Results: Among 37,315 eligible admissions, 22,284 had eligible intake notes; 22,270 produced parseable JSON. In detected-versus-not-detected coding, human-plus-LLM reliability did not differ significantly from human-only reliability across measures (human {kappa} 0.71-0.94; human-plus-LLM {kappa} 0.70-0.93). Stability was associated with human agreement: mean LLM-human agreement increased from 42.6% for classifications with less than 80% stability to 82.7% for classifications with 100% stability (Pearson r = 0.36). Full-cohort extraction showed frequent and overlapping documented factors: sleep disruption was most frequently detected (57.7%), followed by peer conflict (47.2%), academic issues (43.4%), disciplinary issues (43.3%), and attendance issues (16.9%). Discussion: Agreement varied by construct and was strongest when repeated model outputs were stable. Conclusion: Locally hosted open-weight LLMs can support scalable structured extraction of documented psychosocial factors from pediatric psychiatric intake notes after local validation.
Matching journals
The top 3 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- Measuring Quality-of-Care in Treatment of Children with Attention-Deficit/Hyperactivity Disorder: A Novel Application of Natural Language Processing 95%
- Temporally-Informed Random Forests for Suicide Risk Prediction 91%
- Large Language Models Facilitate the Generation of Electronic Health Record Phenotyping Algorithms 91%
Similar papers in this journal
- Evaluation of a Large Language Model to Identify Confidential Content in Adolescent Encounter Notes 94%
- Clinical features and burden of post-acute sequelae of SARS-CoV-2 infection in children and adolescents: an exploratory EHR-based cohort study from the RECOVER program 90%
- Impacts of school closures on physical and mental health of children and young people: a systematic review 89%
Similar papers in this journal
- Electronic Health Record Documentation of Psychiatric Assessments in Massachusetts Emergency Department and Outpatient Settings During the COVID-19 Pandemic 94%
- The Impact of the “Muslim Ban” Executive Order on Healthcare Utilization in Minneapolis-St. Paul, Minnesota 90%
- Low adherence to existing model reporting guidelines by commonly used clinical prediction models 89%
Similar papers in this journal
- The Benefits and Harms of Open Notes in Mental Health: A Delphi Survey of International Experts 93%
- Emergency department visits and boarding for pediatric patients with suicidality before and during the COVID-19 pandemic 93%
- Heterogeneity of Diagnosis and Documentation of Post-COVID Conditions in Primary Care: A Machine Learning Analysis 90%
Similar papers in this journal
- Assessment of machine learning algorithms in national data to classify the risk of self-harm among young adults in hospital: a retrospective study 91%
- Two Data-Driven Approaches to Identifying the Spectrum of Problematic Opioid Use: A Pilot Study within a Chronic Pain Cohort 91%
- A Deep Learning Method to Detect Opioid Prescription and Opioid Use Disorder from Electronic Health Records 89%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.