A point cloud segmentation framework for image-based spatial transcriptomics

Recent progress in image-based spatial RNA profiling enables to spatially resolve tens to hundreds of distinct RNA species with high spatial resolution. It hence presents new avenues for comprehending tissue organization. In this context, the ability to assign detected RNA transcripts to individual cells is crucial for downstream analyses, such as in-situ cell type calling. Yet, accurate cell segmentation can be challenging in tissue data, in particular in the absence of a high-quality membrane marker. To address this issue, we introduce ComSeg, a segmentation algorithm that operates directly on single RNA positions and that does not come with implicit or explicit priors on cell shape. ComSeg is thus applicable in complex tissues with arbitrary cell shapes. Through comprehensive evaluations on simulated datasets, we show that ComSeg outperforms existing state-of-the-art methods for in-situ single-cell RNA profiling and cell type calling. On experimental data, our method also demonstrates proficiency in estimating RNA profiles that align with established scRNA-seq datasets. Importantly, ComSeg exhibits a particular efficiency in handling complex tissue, positioning it as a valuable tool for the community.