Kaiser Permanente National Cross-Vendor Validation of Mammography Artificial Intelligence Computer-Aided Diagnosis Algorithms in a US-Representative Population
Arasu, V.; Gadgil, T.; Westley, M.; Pu, A.; Lee, C.; Smith Gueye, C.; Balkman, J.; Wisner, D.; Ivansco, L.; Kushi, L.
Show abstract
Background Artificial intelligence computer-aided diagnosis (AI CAD) algorithms for screening mammography have shown promise, but independent head-to-head comparisons of commercial algorithms on large, diverse cohorts remain limited. Methods 786,124 mammography screening exam performed from January 1 2022 to December 31 2023 were identified from electronic medical records. Three FDA-cleared commercial algorithms (A, B, D) and the open-source academic model Mirai (C), were applied to the four standard screening views. Breast cancer within 12 months was ascertained through cancer registry linkage. Algorithms were compared by AUROC; sensitivity, specificity, and PPV at matched AI-positive rates of 5%, 10%, 15%, 20%, 25%; and the proportion of false negatives classified as AI-positive. Results Of 707,922 examinations with complete data, 4,436 were associated with cancer. Radiologists recalled 6.8% of examinations, with a sensitivity of 67.2%. Algorithm D had the highest AUROC (0.849), followed by algorithm B (0.818), Mirai (0.817), and algorithm A (0.815). Algorithm D also had the highest sensitivity at every AI-positive rate, from 55.3% (95% CI: 54.1, 56.6) at 5% to 77.9% (95% CI: 76.9, 79.0) at 25%, with CIs that did not overlap those of the other algorithms. No algorithm matched radiologist sensitivity at AI-positive rates of 10% or less. Algorithm B flagged a proportion of radiologist false negatives similar to that of algorithm D from 5% through 20% (20.8% vs 20.5% at 5%), despite its lower standalone sensitivity. Conclusions AI CAD algorithms differed meaningfully in performance on the same large, diverse screening examination cohort, and an open-source academic model performed comparably to two of the three commercial products. Independent local validation at clinically relevant operating points should precede adoption of mammography AI, and algorithm choice and threshold should be matched to the intended workflow.
Matching journals
The top 6 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- Classification performance bias between training and test sets in a limited mammography dataset 93%
- Understanding motivations of older women to continue or discontinue breast cancer screening 91%
- Influence of follow-up, screening age, interval, and compliance on overdiagnosis of ductal carcinoma in situ (DCIS): a modelling study 90%
Similar papers in this journal
- Model uncertainty estimates for deep learning mammographic density prediction using ordinal and classification approaches 94%
- Mammographic density assessed using deep learning in women at high risk of developing breast cancer: the effect of weight change on density 93%
- Breast density prediction from low and standard dose mammograms using deep learning: effect of image resolution and model training approach on prediction quality 90%
Similar papers in this journal
- Evaluation of an artificial intelligence model for detection of pneumothorax and tension pneumothorax on chest radiograph 89%
- Incorporating Polygenic Risk Scores and Nongenetic Risk Factors for Breast Cancer Risk Prediction among Asian Women, Results from Asia Breast Cancer Consortium 89%
- Point-of-care digital cytology with artificial intelligence for cervical cancer screening at a peripheral clinic in Kenya 89%
Similar papers in this journal
- A Machine Learning Ensemble Based on Radiomics to Predict BI-RADS Category and Reduce the Biopsy Rate of Ultrasound-Detected Suspicious Breast Masses 94%
- The NILS study protocol - a retrospective validation study of a preoperative decision-making tool for non-invasive lymph node staging in women with primary breast cancer [ISRCTN14341750] 91%
- Auto-detection of motion artifacts on CT pulmonary angiograms with a physician-trained AI algorithm 90%
Similar papers in this journal
- Diagnostic Accuracy of Artificial Intelligence in Classifying HER2 Status in Breast Cancer Immunohistochemistry Slides and Implications for HER2-Low Cases: A Systematic Review and Meta-Analysis 92%
- Public evidence on AI products for digital pathology 91%
- A human-in-the-loop explanation framework for morphologically transparent AI predictions from whole-slide images 90%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.