AutoML-Multiverse: An Instability-Aware Framework for Quantifying Analytic Variability in Alzheimer's Disease Machine-Learning Studies

Machine-learning (ML) models for Alzheimers disease (AD) frequently yield divergent conclusions, raising concerns about robustness, reproducibility, and interpretability. This instability is partially driven by researcher biases and analytic variability. Coupled with the clinical heterogeneity, mixed pathologies, and cohort differences in AD research, these issues limit the reliability and validity of conclusions from individual models. We introduce AutoML-Multiverse, an instability-aware framework characterising how analytic choices influence ML-based conclusions. The AutoML-Multiverse explores a large space of [~]20,000 analysispipelines and by retaining the full distribution of pipelines, enables direct examination of analytic variability. We evaluate this framework across 20 classification tasks in two independent cohorts studying Alzheimers disease progression (ADNI, N[≤]1,930; NACC, N[≤]1,057), using multiple data modalities: neuroimaging, clinical/cognitive and fluid biomarkers. AutoML-Multiverse performance was equal to or better than non-automated models across all tasks. For example, stable versus progressive mild cognitive impairment (MCI) classification accuracy was 0.68{+/-}0.06 (ADNI) and 0.63{+/-}0.08 (NACC), while AD versus cognitively normal (CN) classification reached 0.97{+/-}0.01 (ADNI). Crucially, each modalitys utility was task- and cohort-dependent: clinical measures dominated diagnostic tasks, whereas imaging better predicted progression, with modality preferences often switching between cohorts, highlighting limited generalisability of single-cohort results. Using the AutoML-Multiverse, we obtained strong classification performance without pre-specifying key model design choices. By reducing analysis-driven variability and explicitly characterising uncertainty, instability-aware evaluation can support the development of more robust and clinically applicable prediction models in AD research. HighlightsO_LIAutoML-Multiverse systematically quantifies analytic instability in clinical ML. C_LIO_LIAnalysis of [~]20,000 pipelines across ADNI and NACC cohorts. C_LIO_LIPipeline choices substantially alter model rankings and biomarker importance. C_LIO_LICross-cohort variability highlights risks of single-dataset studies. C_LIO_LIInstability-aware evaluation improves robustness of AI-driven research. C_LI