Impact of Case and Control Selection on Training AI Screening of Cardiac Amyloidosis

BackgroundRecent studies suggest that cardiac amyloidosis (CA) is significantly underdiagnosed. For rare diseases like CA, the optimal selection of cases and controls for artificial intelligence (AI) model training is unknown and can significantly impact model performance. ObjectivesThis study evaluates the performance of ECG waveform-based AI models for CA screening and assesses impact of different criteria for defining cases and controls. MethodsModels were trained using different criteria for defining cases and controls including amyloidosis by ICD 9/10 code, cardiac amyloidosis, patients seen in CA clinic). The models were then tested on test cohorts with identical selection criteria as well as population-prevalence cohorts. ResultsIn matched held out test datasets, different model AUCs ranged from 0.660 to 0.898. However, these same algorithms exhibited variable generalizability when tested on a population cohort, with AUCs dropping to 0.467 to 0.880. More stringent case definitions during training result in higher AUCs on the similarly constructed test cohort; however representative population controls matched for age and sex resulted in the best population screening performance. ConclusionsAUC in isolation is insufficient to evaluate the performance of a deep learning algorithm, and the evaluation in the most clinically meaningful population is key. Models designed for disease screening are best with matched population controls and performed similarly irrespective of case definitions.