Accuracy and confidence calibration of large language models on neurology board-style questions
Graham, N. S.; Auger, S. D.; Attaallah, B.; Batla, A.; Scott, G.
Show abstract
Large language models (LLMs) are increasingly evaluated for clinical tasks, but safe use requires reliable uncertainty estimates as well as high accuracy. We tested eight general-purpose LLMs on 100 validated neurology board-style multiple-choice questions and compared three confidence signals: self-reported confidence, critique-implied confidence from an independent model instance, and consistency-based confidence from repeated sampling. Accuracy varied from 45% (Llama 3.1 8B) to 92% (Gemini 3 Flash/Pro), but confidence was generally overestimated, particularly in lower-accuracy models. Consistency-based confidence provided the best discrimination between correct and incorrect answers (pooled AUROC 0.74), outperforming self-report, although calibration differences between methods were not significant. Calibration should therefore be reported alongside accuracy when clinical LLMs are evaluated.
Matching journals
The top 5 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- From Tool to Teammate: A Randomized Controlled Trial of Clinician-AI Collaborative Workflows for Diagnosis 93%
- A Framework to Assess Clinical Safety and Hallucination Rates of LLMs for Medical Text Summarisation 91%
- Enhancing Privacy-Preserving Deployable Large Language Models for Perioperative Complication Detection: A Targeted Strategy with LoRA Fine-tuning 90%
Similar papers in this journal
- Is One Run Enough? Reproducibility of Flagship Large Language Models Across Temperature and Reasoning Settings in Biomedical Text Processing 91%
- Generative Large Language Models in Electronic Health Records for Patient Care Since 2023: A Systematic Review 91%
- Large Language Models Facilitate the Generation of Electronic Health Record Phenotyping Algorithms 91%
Similar papers in this journal
- Enhancing Early Detection of Cognitive Decline in the Elderly through Ensemble of NLP Techniques: A Comparative Study Utilizing Large Language Models in Clinical Notes 91%
- irAE-GPT: Leveraging large language models to identify immune-related adverse events in electronic health records and clinical trial datasets 91%
- Consistent Performance of GPT-4o in Rare Disease Diagnosis Across Nine Languages and 4967 Cases 91%
Similar papers in this journal
- Comparison of local large language models for extraction of signs and symptoms data from electronic health records 90%
- Optimising supervised machine learning algorithms predicting cigarette cravings and lapses for a smoking cessation just-in-time adaptive intervention (JITAI) 89%
- Protocol For Human Evaluation of Artificial Intelligence Chatbots in Clinical Consultations 89%
Similar papers in this journal
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.