Automated Interpretation of EEG Reports Using a Large Language Model with Structured Confidence Outputs

Background: Free-text EEG reports typically lack structure, hindering scalable analysis. We evaluate a large language model (LLM) pipeline to extract structured diagnostic labels and confidence levels from these reports. Methods: We developed a hierarchical annotation schema to classify EEG reports for four specific abnormality types using a four-point confidence scale. To establish ground truth, two certified EEG technicians annotated a diverse dataset of reports authored by neurologists with distinct writing styles. We then implemented a grammar-constrained Mistral-7B pipeline, iteratively prompt-tuned on a development set to mirror these expert annotations. The pipeline's effectiveness was evaluated against the human expert benchmark using core agreement (diagnostic accuracy) and certainty-adjusted agreement (confidence alignment), with classical NLP models serving as a secondary baseline. Results: Mistral-7B significantly outperformed baselines, achieving 96% accuracy for overall abnormality detection, approaching the human benchmark of 98%. Crucially, the model successfully identified rare epileptiform abnormalities where traditional models failed and generalized robustly across distinct reporting styles. While diagnostic accuracy was high, a performance gap persisted in certainty-adjusted agreement, indicating that accurately modeling nuanced clinical confidence remains a challenge. Conclusion: LLMs can effectively automate the extraction of structured diagnostic information from EEG reports with near-human accuracy and strong generalization. While confidence calibration requires further refinement, the combination of accurate classification and explainability makes this pipeline a promising tool for standardizing clinical data at scale. Keywords: Routine Clinical Electroencephalography; Large Language Models; Clinical NLP; Confidence Assessment; Explainable AI; Neurophysiological Evaluation