Self-reported diagnoses as training labels for finding uncoded chronic fatigue, fibromyalgia, and endometriosis in the All of Us Research Program

Objective: Case-finding models for underdiagnosed conditions are usually trained on diagnosis codes, which are missing from the records of many patients with a clinical diagnosis. We asked whether models trained on self-reported diagnoses rank uncoded self-reporters higher than models trained on later codes, and how a score fit only from released summary counts compares with a participant-level model. Methods: In the All of Us Research Program, we followed survey respondents with no qualifying diagnosis code by the survey date for later codes. For chronic fatigue, fibromyalgia, and endometriosis (participants recorded as female at birth), we fit a naive Bayes score with empirical-Bayes shrinkage, from released feature-by-label counts for the self-reported diagnosis and from exact counts inside the data enclave for a later code, and an elastic net on participant-level data for each label. In a random 20% validation split, we measured discrimination of self-report, of uncoded self-reporters (no qualifying code observed), and of later codes. Results: For chronic fatigue (149,860 respondents, including 7,770 self-reporters, 95% with no code observed), the summary-count score trained on self-report ranked uncoded self-reporters with an area under the receiver operating characteristic curve (AUROC) of 0.74 (95% CI, 0.72-0.75), against 0.67 (95% CI, 0.65-0.69) when trained on later codes. For later codes, the two scores were similar (0.69 vs 0.70). Fibromyalgia showed the same pattern (0.73 vs 0.64 and 0.74 vs 0.72). Elastic nets trained on self-report ranked uncoded self-reporters higher than the summary-count scores (0.80 vs 0.74 and 0.78 vs 0.73) but discriminated later codes less well than code-trained elastic nets. For endometriosis, the self-report score also ranked uncoded self-reporters above the code-trained score (0.61 vs 0.53), but discrimination was low and it did not discriminate later codes (0.50 vs 0.70). Conclusion: The choice of training label changes which patients a case-finding model ranks highly. For all 3 conditions, self-reported diagnoses were the more useful label for finding uncoded self-reporters. For chronic fatigue and fibromyalgia, the self-report score also discriminated later codes as well as the code-trained score. Independent clinical validation is needed before these models guide case finding.