Expert Discrimination of AI-Generated versus Authentic Radiologic Images: A Multimodal, Pre-Registered Visual Turing Test
Nam, Y.; An, T.; Hwang, S. I.; Jang, H.; Jeon, C.; Jeong, J. W.; Jeong, J.; Kim, D. Y.; Kim, S. Y.; Kim, S.; Kim, Y.; Lee, K. H.; Oh, H. S.; Park, J. H.; Seo, M.; Sim, Y.; Song, J. M.; Song, S.; Yoon, H. M.; the MeducAI Reader Study Group, ; Hong, P.; Kim, N.
Show abstract
Background: Frontier text-to-image models can synthesise radiologic images of high realism, raising the question of whether expert radiologists can serve as a provenance safeguard for the medical image record. Methods: We conducted a prospective, pre-registered visual Turing test in which 60 invited Korean board-certified radiology faculty and trainees judged authentic (teaching-repository) and AI-generated radiologic images from a locked pool of 241 displayable cells (82 entities; nine subspecialties; six modalities; 60 readers x 60 trials = 3,600 reader-image observations) produced by two contemporary commercial generators. The primary endpoint was the confidence-weighted, reader-averaged multi-reader multi-case area under the curve for AI versus authentic images, conditional on the locked image pool; the key secondary endpoint was the Faculty-minus-Junior difference under a two one-sided tests equivalence framework. The pre-specified statistical analysis plan was registered on the Open Science Framework before data lock. Findings: All 60 readers completed the test. The pooled confidence-weighted area under the curve was 0.71 (95% CI, 0.69 to 0.74), above the null value of 0.5 but within the pre-specified modest tier (0.60 to 0.75). The Faculty-minus-Junior contrast was 0.04 (95% CI, -0.02 to 0.10), including zero, and the two one-sided tests established equivalence within the +/-0.10 margin. No reader stratum and no pre-specified sensitivity analysis reached the deployable-classifier threshold (area under the curve >= 0.75). Interpretation: In this single-country cohort, expert radiologists distinguished frontier-generated from authentic radiologic images only modestly, without a meaningful expertise gradient (equivalence within +/-0.10) and with no reader stratum reaching a standalone provenance safeguard. These findings support radiology AI-literacy training and pipeline-level provenance safeguards rather than reliance on reader judgment, and warrant retesting in an independent reader cohort. Funding: This research was supported by a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (grant number: RS-2025-02213531).
Matching journals
The top 7 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- Development and validation of AI-based pre-screening of large bowel biopsies 92%
- CARDBiomedBench: A Benchmark for Evaluating Large Language Model Performance in Biomedical Research 90%
- Novel deep learning algorithm predicts the status of molecular pathways and key mutations in colorectal cancer from routine histology images 88%
Similar papers in this journal
- Segmenting functional tissue units across human organs using community-driven development of generalizable machine learning algorithms 92%
- Integration of clinical characteristics, lab tests and a deep learning CT scan analysis to predict severity of hospitalized COVID-19 patients 91%
- The Impact of Digital Histopathology Batch Effect on Deep Learning Model Accuracy and Bias 91%
Similar papers in this journal
Similar papers in this journal
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.