Disagreement between AI and Radiologists: Quantifying Direction and Fragility
El Kaddouri, B.; Gevenois, P. A.; Bankier, A. A.
Show abstract
Background: Conventional performance metrics do not indicate whether artificial intelligence (AI)-platforms and radiologists are right or wrong on the same examinations, nor whether their non-agreement is more often beneficial or harmful. Objective: To develop, illustrate, and apply directional metrics quantifying non-agreement between radiologists and AI-platforms against a common reference standard, and to adapt a fragility-based framework for quantifying the statistical robustness of these metrics. Methods: We defined Positive Non-Agreement (PNA, AI-platform correct, radiologist incorrect), Negative Non-Agreement (NNA, radiologist correct, AI-platform incorrect), and Net Non-Agreement (NENA, their difference). Metrics were illustrated in three hypothetical scenarios of 1,000 examinations each and applied to three previously published AI-radiologist studies on pulmonary embolism and mammography. Bootstrap confidence intervals were calculated, and a reclassification fragility index (FI) quantified the minimum case reclassification needed to remove statistical support for NENA's direction. Results: In illustration, NENA was positive in two scenarios and negative in one. In the three published studies, NENA ranged from -4.18% to +10.74% and decreased consistently with increasing radiologist experience (junior residents, +6.27%; consultants, +1.45%), a gradient driven primarily by declining PNA with modest increase in NNA. The reclassification FI was defined for all three illustrative scenarios (range, 0.10%-1.70%) and for six of the seven comparisons from these studies (range, 0.34%-4.31%), but not for the weakest comparison, whose confidence interval for NENA included zero. Conclusions: PNA, NNA, and NENA provide a directional, case-level complement to conventional performance metrics for AI-radiologist non-agreement. The accompanying reclassification FI offers a statistically grounded, generalizable method for quantifying their robustness.
Matching journals
The top 5 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- Classification performance bias between training and test sets in a limited mammography dataset 92%
- A method for rapid machine learning development for data mining with Doctor-In-The-Loop 91%
- Opening the Black Box of Artificial Intelligence for Clinical Decision Support: A Study Predicting Stroke Outcome 91%
Similar papers in this journal
- Artificial Intelligence for Surgical Scene Understanding: A Systematic Review and Reporting Quality Meta-Analysis 91%
- A typology of physician input approaches to using AI chatbots for clinical decision-making: a mixed methods study 91%
- A human-in-the-loop explanation framework for morphologically transparent AI predictions from whole-slide images 91%
Similar papers in this journal
- Assessing GPT-4 Multimodal Performance in Radiological Image Analysis 94%
- Evaluating Large Language Model-Generated Brain MRI Protocols: Performance of GPT4o, o3-mini, DeepSeek-R1 and Qwen2.5-72B 91%
- Observer agreement and clinical significance of chest CT reporting in patients suspected of COVID-19 90%
Similar papers in this journal
- Theory of radiologist interaction with instant messaging decision support tools: a sequential-explanatory study 93%
- Designing a computer-assisted diagnosis system for cardiomegaly detection and radiology report generation 93%
- Cardiology Knowledge Assessment of Retrieval-Augmented Open versus Proprietary Large Language Models 92%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.