Underdiagnosis Bias of Chest Radiograph Diagnostic AI can be Decomposed and Mitigated via Dataset Bias Attributions

Inequitable diagnostic accuracy is a broad concern in AI-based models. However, current characterizations of bias are narrow, and fail to account for systematic bias in upstream data-collection, thereby conflating observed inequities in AI performance with biases due to distributional differences in the dataset itself. This gap has broad implications, resulting in ineffective bias-mitigation strategies. We introduce a novel retrospective model evaluation procedure that identifies and characterizes the contribution of distributional differences across protected groups that explain population-level diagnostic disparities. Across three large-scale chest radiography datasets, we consistently find that distributional differences in age and confounding image attributes (such as pathology type and size) contribute to poorer model performance across racial subgroups. By systematically attributing observed underdiagnosis bias to distributional differences due to biases in the data-acquisition process, or dataset biases, we present a general approach to disentangling how different types of dataset biases interact and compound to create observable AI performance disparities. Our method is actionable to aid the design of targeted interventions that recalibrate foundation models to specific subpopulations, as opposed to methods that ignore systematic contributions of upstream data biases on inequitable AI performance.