Evaluating the Interplay and Impact of Quality and Technical Variations on Performance of AI Models in Multi-Institutional Prostate MRI Datasets

Purpose: To evaluate whether visually assessed high-quality prostate MRI, as defined by the Prostate Imaging Quality (PI-QUAL) scoring system, is sufficient to ensure consistent AI model performance, or whether hidden acquisition-related technical effects introduce variability that cannot be detected through visual inspection. Methods: This retrospective study included 568 bpMRI scans from seven institutions, classified using PI-QUAL v2 into high-quality (scores 2-3) and low-quality (score 1) groups. Quantitative image quality features were extracted using MRQy from T2w and ADC sequences. Three experiments were conducted on high-quality scans: (1) unsupervised clustering to identify hidden technical batch effects; (2) V-Net models (M) for prostate capsule segmentation on T2w using single-cluster and balanced multi-cluster training strategies; and (3) VoxelMorph-based deformable registration (M2) to align ADC maps to T2w images. Performance metrics included DSC, precision, recall, volume ratio, HD95, ASSD, and TRE. Kruskal-Wallis tests assessed cross-cluster variability, Two One-Sided Tests (TOST) evaluated performance equivalence, and bootstrap resampling quantified training efficiency. Results: Clustering identified four clusters among 277 high-quality bpMRI scans. Single-cluster models showed variable performance (DSC=0.41-0.85) with significant batch effects (p<0.001), even with 140 training samples. The balanced multi-cluster model (n=60) eliminated batch effects (p=0.075), achieved equivalent performance (p<0.001), and demonstrated 2.23-fold greater training efficiency despite using 2.3-fold fewer samples. Registration performance also varied significantly across clusters (TRE=0.86-5.47 mm, p<0.001). Conclusion: Acquisition-related batch effects significantly impact AI generalizability in prostate MRI, even among high-quality scans. Quantitative clustering enables detection of technical variations and supports diversity-aware training strategies that achieve equivalent performance with 2.3-fold fewer samples while eliminating batch effects.