AI-assisted identification of potential treatment injury from inpatient discharge summaries: an early-stage sociotechnical evaluation in a New Zealand district hospital

Objectives: To assess the feasibility and early performance of an AI-assisted screening workflow that reviews inpatient discharge summaries to identify potential treatment injuries for clinician review and, where appropriate, Accident Compensation Corporation (ACC) treatment injury claim consideration in New Zealand. Design: Single-site, six-month pilot reported as a quality-improvement/service-evaluation activity. A large language model (LLM)-enabled system acted as an index screening test over discharge summaries to generate candidate treatment injury signals for structured clinician adjudication and escalation. Setting: A New Zealand district hospital (Wairarapa). Screening operated on routinely produced inpatient discharge summaries. Participants: All inpatient discharge summaries produced during the pilot period were screened (n = 2,642). The unit of analysis was one discharge summary per inpatient episode. Intervention: An LLM-based screening workflow that outputs structured candidate harm signals with supporting excerpts and a certainty label. Outputs were reviewed by clinicians using a structured adjudication template. Suitable cases could be escalated to senior clinicians and followed through the local ACC treatment injury process. Main outcome measures: Primary outcomes were yield, positive predictive value (PPV) of AI- flagged signals for clinician-adjudicated iatrogenic harm, and the rate of unsupported/hallucinated outputs. Secondary outcomes included overlap with incident reporting and administrative coding, downstream ACC claim outcomes, and clinician review workload. Results: Across 2,642 inpatient discharge summaries screened, the AI-assisted workflow generated 237 candidate signals (yield 9.0%; 95% CI 7.9% to 10.1%). Clinician adjudication classified 102 as confirmed iatrogenic harm, 43 as possible harm requiring additional context, 81 as no harm, and 11 as unsupported/hallucinated outputs. PPV for confirmed harm was 43.0% (95% CI 36.8% to 49.4%). PPV for confirmed-or-possible harm was 61.2% (95% CI 54.9% to 67.1%). Seven ACC treatment injury claims were lodged as a direct result of AI-assisted surfacing; three were accepted, two declined, and two were held/pending at analysis. Organisational ACC data identified 68 treatment injury claims lodged during the same period, of which 17 overlapped with AI-identified cases. Conclusions: AI-assisted screening of discharge summaries can feasibly increase detection of potential treatment injuries for structured clinician review and may support more systematic ACC consideration and safety learning. Safe scaling requires transparent reporting of model and prompt details, robust human adjudication and escalation pathways, and governance controls to mitigate false positives and unsupported outputs. A next-stage evaluation should estimate diagnostic accuracy against a chart-review reference standard and examine implementation outcomes including workload, acceptability, fidelity and sustainability.