Beyond the Final Layer: Evaluating Single-Cell Foundation Model Representations for Molecular Prediction
Wen, M.-C. C.; Lyu, Y.; Robbins, A.
Show abstract
Single-cell foundation models (scFMs) are commonly benchmarked using final-layer embeddings, although useful depth may depend on the task. We evaluate Geneformer, scBERT, scGPT, and scFoundation as frozen feature extractors for cell-type and perturbation-response prediction using layer-wise probes, leave-one-dataset-out (LODO) selection, readout controls, representation diagnostics, centered kernel alignment (CKA), and retrieval/collapse analyses. Intermediate layers significantly improve cell-type accuracy for scGPT and scBERT in several settings, while Geneformer and scFoundation generally show performance comparable to their final layers. For perturbation prediction, LODO identifies layer 1 as an earliest-on-par candidate across all four models, without establishing universal superiority. Paired scGPT CLS and mean-pooling analyses show that readout affects preferred depth. CKA and representation metrics reveal model- and task-dependent geometric changes, while retrieval and collapse checks support the robustness of perturbation findings. We recommend reporting extraction depth, readout, selection protocol, and task-specific diagnostics. A validated non-final layer may also reduce computation, although efficiency was not directly benchmarked.
Matching journals
The top 8 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- Benchmarking computational methods for multi-omics biomarker discovery in cancer 93%
- Scalable embedding fusion with protein language models: insights from benchmarking text-integrated representations 93%
- PLMFit : Benchmarking Transfer Learning with Protein Language Models for Protein Engineering 92%
Similar papers in this journal
- Capturing cell heterogeneity in representations of cell populations for image-based profiling using contrastive learning 93%
- Predicting drug polypharmacology from cell morphology readouts using variational autoencoder latent space arithmetic 93%
- A novel transformer-based platform for the prediction and design of biosynthetic gene clusters for (un)natural products 92%
Similar papers in this journal
Similar papers in this journal
Similar papers in this journal
- DrDimont: Explainable drug response predictionfrom differential analysis of multi-omics networks 92%
- Interpretable deep learning architectures for improving drug response prediction performance: myth or reality? 92%
- Unsupervised protein embeddings outperform hand-crafted sequence and structure features at predicting molecular function 92%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.