Calibration, Uncertainty, and Data Efficiency of Deep Learning Chest Radiograph Classifiers Under Cross-Dataset Distribution Shift: A Multi-Dataset Benchmark
Shamim, F.; Hussain, R. A.
Show abstract
Purpose To evaluate the external transportability of source-derived probability calibration for chest radiograph deep learning classifiers under distribution shift and determine its dependence on sample size and disease prevalence. Materials and Methods This retrospective secondary computational analysis of public, de-identified radiographic datasets involved no participant contact or new data collection. Cohort sizes were prespecified by availability without prospective power calculation: 106,686 NIH ChestX-ray14 images (29,323 patients; 1992-2015) for development and internal testing, 20,000 CheXpert images (20,000 patients; October 2002-July 2017), and 14,998 VinBigData images (2018-2020) for external testing. ResNet-18, DenseNet-121, and DeiT-Tiny classifiers for cardiomegaly and pleural effusion were trained across four patient fractions and three seeds (36 fits). Calibrators were fitted exclusively on NIH data. Twelve primary external Brier-score contrasts (temperature-scaled versus raw probabilities) were evaluated using paired cluster-bootstrapping with Holm adjustment. Results Temperature scaling reduced Brier scores in nine of 12 primary contrasts but worsened pleural effusion calibration on VinBigData across all architectures (Brier increases: ResNet-18, 0.001460 [95% CI: 0.001347, 0.001568]; DenseNet-121, 0.001193 [95% CI: 0.001109, 0.001274]; DeiT-Tiny, 0.000498 [95% CI: 0.000417, 0.000577]; all adjusted P=.006). Standardizing class weights to NIH prevalence reversed all CheXpert improvements. Although full versus 10% training improved external AUROC across all comparisons, DeiT-Tiny CheXpert pleural effusion Brier score worsened by 0.007039 despite a 0.032466 AUROC gain. At 80% selective image retention, calibrated ensembles retained only 12.3%-17.2% of VinBigData positive pleural effusion examinations. Conclusion Source-fitted calibration is not uniformly transportable across health systems, and discrimination gains do not guarantee calibrated probabilities. External evaluation must simultaneously assess probability reliability, prevalence shifts, and retained-case composition.
Matching journals
The top 7 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- Development and validation of AI-based pre-screening of large bowel biopsies 90%
- Multicenter Validation of a Machine Learning Algorithm for Diagnosing Pediatric Patients with Multisystem Inflammatory Syndrome and Kawasaki Disease 88%
- X-ray Dark-Field Chest Imaging can Detect and Quantify Emphy-sema in COPD Patients 88%
Similar papers in this journal
- Machine Learning Generalizability Across Healthcare Settings: Insights from multi-site COVID-19 screening 91%
- CT-based Rapid Triage of COVID-19 Patients: Risk Prediction and Progression Estimation of ICU Admission, Mechanical Ventilation, and Death of Hospitalized Patients 90%
- How AI is used in FDA-authorized medical devices: a taxonomy across 1,016 authorizations 90%
Similar papers in this journal
- Enhancing Semantic Segmentation in Chest X-Ray Images through Image Preprocessing: ps-KDE for Pixel-wise Substitution by Kernel Density Estimation 90%
- Machine learning in predicting respiratory failure in patients with COVID-19 pneumonia - challenges, strengths, and opportunities in a global health emergency 90%
- Deep learning models for COVID-19 chest x-ray classification: Preventing shortcut learning using feature disentanglement 90%
Similar papers in this journal
- Integration of clinical characteristics, lab tests and a deep learning CT scan analysis to predict severity of hospitalized COVID-19 patients 92%
- Weakly supervised classification of rare aortic valve malformations using unlabeled cardiac MRI sequences 90%
- Artificial Intelligence System Reduces False-Positive Findings in the Interpretation of Breast Ultrasound Exams 89%
Similar papers in this journal
- Development and Validation of a Deep Learning Model for Detecting Signs of Tuberculosis on Chest Radiographs among US-bound Immigrants and Refugees 94%
- Designing a computer-assisted diagnosis system for cardiomegaly detection and radiology report generation 92%
- Modular Clinical Decision Support Networks (MoDN)—Updatable, Interpretable, and Portable Predictions for Evolving Clinical Environments 90%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.