Prospective validation of an uncertainty-aware machine learning model for secondary progressive multiple sclerosis classification

Background: Relapsing-remitting multiple sclerosis (RRMS) often progresses to secondary-progressive MS (SPMS), but diagnosis is typically retrospective, delaying potential interventions. Existing rule-based and machine learning approaches often lack prospective validation and provide no uncertainty estimates, limiting clinical adoption. Objective: To prospectively evaluate a conformal prediction (CP)-enabled random forest model for reproducing clinician-recorded disease course labels in the Swedish Multiple Sclerosis Registry (SMSReg) and compare its classifications with the Lorscheider criteria. Methods: The CP model was trained and calibrated on SMSReg-2022 (10,787 individuals, 85,461 hospital visits) and then validated on SMSReg-2025 (15,719 individuals, 45,077 hospital visits; not included in the SMSReg-2022 training/calibration/testing datasets). Performance was assessed at the hospital visit and individual levels and compared with the Lorscheider criteria. Results: The model maintained good calibration on the prospective dataset. At the per-hospital visit-level, the model accuracies were 92% (RRMS) and 95% (SPMS). At the individual level, sensitivity was 98%, specificity 89%, and F1-score 0.75. With clinician-recorded labels as reference, the model showed higher sensitivity than the Lorscheider criteria (98% vs. 71%), more transition assignments (90% vs. 57%), and higher F1-score (0.76 vs. 0.69). Conclusion: The CP-enabled model prospectively reproduced clinician-recorded disease course labels with calibrated uncertainty, supporting harmonization and decision support.