Robustness to nuisance perturbations enables unsupervised evaluation of single-cell foundation models
Sallam, A.; Gillis, J.
Show abstract
Single-cell foundation model evaluations have relied almost exclusively on downstream tasks. While these tasks measure whether an embedding recovers annotated cell types, batches, or trajectories, they cannot determine if that structure is reproducible or merely an artifact of a single noisy draw, a key limitation since incomplete sampling is intrinsic to single-cell measurement. Here, we introduce a fully unsupervised evaluation framework grounded in a fundamental principle: a faithful representation must preserve its neighbourhood structure under nuisance perturbations that mimic technical and sampling variation. Across five scFMs, a PCA baseline, and 39 datasets, we show that models ranked as near-equivalent by standard benchmarks differ nearly twofold in local neighbourhood preservation under a perturbation discarding just 5% of counts. This structural instability is scale-dependent and often masked by visually coherent embeddings. Cluster-level stability under resampling tracks established bio-conservation metrics (Spearman {rho}=0.78), showing that invariance to nuisance perturbations captures representation quality no benchmark measures directly.
Matching journals
The top 5 journals account for 50% of the predicted probability mass.
Similar papers in this journal
Similar papers in this journal
- Learning multi-cellular representations of single-cell transcriptomics data enables characterization of patient-level disease states 94%
- Single-cell morphodynamical trajectories enable prediction of gene expression accompanying cell state change 94%
- Inferring high-dimensional pathways of trait acquisition in evolution and disease 93%
Similar papers in this journal
- Computing the Riemannian curvature of image patch and single-cell RNA sequencing data manifolds using extrinsic differential geometry 93%
- Constrained Evolutionary Funnels Shape Viral Immune Escape 93%
- Predicting clinical drug response from model systems by non-linear subspace-based transfer learning 93%
Similar papers in this journal
- An in-depth comparison of linear and non-linear joint embedding methods for bulk and single-cell multi-omics 94%
- SHEST: Single-cell-level artificial intelligence from haematoxylin and eosin morphology for cell type prediction and spatial transcriptomics reconstruction 92%
- Scalable embedding fusion with protein language models: insights from benchmarking text-integrated representations 92%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.