ClustSIGNAL identifies cell types and subtypes using an adaptive smoothing approach for scalable spatial clustering

The increased uptake of high-resolution spatially-resolved transcriptomics (SRT) technologies demands the development of unsupervised methods to extract cell types and their spatial distribution from biological tissues. However, unsupervised clustering is challenging due to the sparsity of the data and the differences in cell arrangement within tissues. Here, we introduce ClustSIGNAL, a spatial clustering method that adaptively uses neighbourhood information to overcome data sparsity and perform cell type clustering. ClustSIGNAL first defines initial clusters and sub-clusters of cells with similar gene expression patterns. For each cell, a fixed neighbourhood size is defined, and entropy is calculated based on the proportion of initial subclusters in the neighbourhood to capture its composition. Cell-specific weights, generated from entropy values, are used to embed spatial information into the gene expression through adaptive smoothing. The transformed gene expression is then used for clustering cell types. We compared our adaptive smoothing approach with other smoothing scenarios on four simulated datasets of varying spatial complexity. We also evaluated our clustering method on four publicly available high-resolution SRT datasets and compared its performance to that of three other spatial clustering methods. We showed that ClustSIGNAL performs multi-sample clustering with high accuracy and can identify subtle cell types and subtypes of biological relevance. It is also robust to changes in spatial structure of tissues, segmentation errors, and sparsity. Overall, ClustSIGNAL stabilises gene expression of cells in homogeneous neighbourhoods and preserves distinct gene expression of cells in heterogeneous regions, effectively balancing the use of neighbouring cells as prior knowledge for downstream analysis. The ClustSIGNAL R/Bioconductor package is available from bioconductor.org/packages/clustSIGNAL.