Quality and transcript-grounded clinical-risk error burden of AI-generated vs allied health professional clinical notes: a paired simulation study across four disciplines
Bergman, H. I.; Liu, V.; Ali, S.
Show abstract
Background: Ambient artificial-intelligence (AI) scribes are being deployed beyond medicine into allied-health practice, but comparative evidence mainly concerns physician documentation. Objective: To compare quality and transcript-grounded clinical-risk error burden of AI-generated and allied health professional notes from identical simulated consultations. Methods: Paired simulation across dietetics, physiotherapy, occupational therapy and speech and language therapy (five consultations each; 20 pairs; 40 notes). Heidi generated an unedited first-draft note and a professional documented each encounter. Professionals rated PDQI-9 quality. Professional and automated review identified unsupported statements and omissions; two language models graded clinical risk. Co-primary outcomes were PDQI-9 score and Critical+High error density. Results: Mean PDQI-9 scores were 38.45 and 35.05 for AI and professional notes (paired difference +3.40, 95% CI +0.01 to +6.79; Wilcoxon p=0.042; dz=0.47). The Bayesian estimate was +3.47 (94% equal-tailed credible interval +0.66 to +6.30; P[AI>professional]=0.989). Automated-reviewer all-error density was 3.35 vs 8.55 per note (professional-minus-AI difference 5.20, 95% CI 3.85 to 6.55); omission density was 1.45 vs 5.20 (difference 3.75, 2.80 to 4.65). Critical+High density was 0.35 vs 0.75 (difference 0.40, -0.10 to 1.00). The pooled quality result was driven principally by dietetics. Conclusions: AI-generated notes achieved modestly higher pooled quality and fewer transcript-grounded errors, chiefly omissions; serious-error evidence was inconclusive. Small discipline samples and incomplete ground truth preclude safety or discipline-wide superiority claims.
Matching journals
The top 7 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- A typology of physician input approaches to using AI chatbots for clinical decision-making: a mixed methods study 94%
- Ambient scribe in general practice: a multi-perspective before-after longitudinal mixed-methods study 93%
- Utilization of Generative AI-drafted Responses for Managing Patient-Provider Communication 92%
Similar papers in this journal
- What Do Clinicians Edit in Ambient AI-Drafted Clinical Documentation? A Qualitative Content Analysis 94%
- Sociodemographic Bias in Large Language Model Clinical Trial Screening 91%
- Beyond Metrics to Methods: A Scoping Review of Large Language Models for Detection of Social Drivers of Health in Clinical Notes 91%
Similar papers in this journal
- Feasibility trial of a new digital training package to enhance primary care practitioners' communication of clinical empathy and realistic optimism 92%
- Clinical academic research in the time of Corona: a simulation study in England and a call for action 92%
- Development of the Tool for Advancing Practice Performance, a practice-level survey to assess primary care structures and processes 91%
Similar papers in this journal
- Use and appropriateness of Reporting Guidelines in physical therapy research: a protocol for a meta-research study 92%
- Large language models for conducting systematic reviews: on the rise, but not yet ready for use – a scoping review 91%
- Clinical practice guidelines and recommendations in the context of the COVID-19 pandemic: systematic review and critical appraisal 90%
Similar papers in this journal
- AI-Generated Clinical Summaries: Errors and Susceptibility to Speech and Speaker Variability 92%
- Connecting Artificial Intelligence and Primary Care Challenges: Findings from a Multi-Stakeholder Collaborative Consultation 90%
- Cracking the Code: A Scoping Review to Unite Disciplines in Tackling Legal Issues in Health Artificial Intelligence 89%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.