An independent, multi-timepoint evaluation of Disconnection Symptom Discoverer cognitive outcome prediction accuracy in stroke

The Disconnection Symptom Discoverer (DSD) model proposes to predict long-term performance on neuropsychological tests from stroke lesion disconnection profiles. The model requires external validation to determine reproducibility and generalizability to new and different patients. Here, we investigated whether the DSD supports accurate multi-domain cognitive outcome predictions at three different timepoints post stroke, in a clinically representative independent cohort. In this study, the DSD was used to predict visuospatial attention, verbal memory, and language scores in an independent cohort of 74 stroke survivors (mean age = 69.2, 39% female) with 3 repeated cognitive assessments. DSD-predicted scores were compared to observed neuropsychological scores collected at <2 weeks, six months, and > 2 years post-stroke. DSD-predicted language outcomes were significantly correlated with observed behaviour at the <2 weeks timepoint, but no other significant correlations between DSD-predicted scores were identified. Importantly, DSD-predicted verbal memory and visuospatial domain scores were not significantly correlated with observed behaviour at any of the considered timepoints (minimum p-value = 0.33). Across all tests and timepoints, DSD-predicted scores had an average Mean Absolute Error (MAE) of 0.21 (SD = 0.13, range = 0.04-0.43), with the highest errors occurring between predicted and observed memory scores. Larger stroke lesions were associated with higher MAE, indicating that the DSD performance was modulated by stroke severity. Overall, these results indicate that the DSD did not yield informative predictions of long-term cognitive outcomes in this external dataset. This finding provides an important illustration of potential overfitting issues within cognitive outcome prediction models, highlighting the need for caution when aiming to predict long-term post-stroke cognitive outcomes and further external validation of proposed models.