A Systematic Evaluation of Single-Cell Batch Integration Metrics and sBEE: A Robust New Metric

Single-cell RNA sequencing (scRNA-seq) datasets generated across laboratories and experimental conditions often exhibit batch effects that obscure biological variation. Numerous computational methods for batch integration have been developed, making rigorous benchmarking critical. Evaluation metrics are central to assessing method performance; however, existing metrics capture only partial aspects of integration quality and often rely on implicit assumptions about cell distributions in the embedding space. Consequently, benchmarking studies frequently report discordant rankings of batch integration methods across metrics, complicating interpretation and method selection. Here, we systematically evaluate widely used metrics under controlled scenarios that isolate common integration challenges, including imbalanced batch composition, partial cell-type overlap, and varying cluster geometries. By stress-testing metrics under these scenarios, we identify the conditions under which each metric succeeds or fails. Based on these observations, we introduce sBEE (single-cell Batch Effect Evaluator), a unified metric that jointly evaluates cross-batch distance relationships and local neighborhood batch composition. Across diverse scenarios, sBEE provides stable assessments of mixing quality and remains robust to failure modes that affect existing metrics. Together, our work provides a systematic evaluation of batch integration metrics and introduces a unified metric for a more reliable assessment of integration quality. Code and datasets are available at https://github.com/tastanlab/sBEE.