A Method for Sensitivity Analysis of Automatic Contouring Algorithms Across Different MRI Contrast Weightings Using SyntheticMR

BackgroundCurrently, a majority of institution-specific automatic MRI-based contouring algorithms are trained, tested, and validated on one contrast weighting (i.e., T2-weighted), however their actual performance within this contrast weighting (i.e., across different repetition times, TR, and echo times, TE) is under-investigated and poorly understood. As a result, external institutions with different scan protocols for the same contrast weighting may experience sub-optimal performance. PurposeThe purpose of this study was to develop a method to evaluate the robustness of automatic contouring algorithms to varying MRI contrast weightings. MethodsOne healthy volunteer and one patient was scanned using SyntheticMR on the MR-Simulation device. The parotid and submandibular glands in these subjects were contoured using an automatic contouring algorithm trained on T2-weighted MRIs. For ground truth manual contours, two radiation oncology residents and one pre-resident physician were recruited and their STAPLE consensus was determined. A total of 216 different MRI TR and TE combinations were simulated across T1-, T2-, and PD-weighted contrast ranges using SyntheticMRs post-processing software, SyMRI. Comparisons between automatic contouring algorithm contours and the ground truth were determined using the Dice similarity coefficient (DSC) and 95th percentile Hausdorff distance (HD95). ResultsNotable differences in the automatic contouring models performance were seen across the contrast-weighted range, even within the T2-weighted range. Further, some models even performed as well or better across subsets of the T1-weighted range. The PD-weighted range saw the worst performance. The range of discrepancy in DSC and HD95 exceeded 0.2 and 3.66 mm, respectively, in some structures. In the T2-weighted contrast region where the model was trained, 100%, 40%, 24%, and 57% for the DSC in the left parotid, right parotid, left submandibular, and right submandibular gland, respectively, exceeded interobserver variability. ConclusionsThis study demonstrates the variable performance of MRI-based automatic contouring algorithms across varying TR and TE combinations. This methodology could be applied in future studies as a method for evaluating model sensitivity, out of distribution detection ability, and performance drift.