SpotDMix: informed mRNA transcript assignment using mixture models

Unveiling the genetic profiles of spatially distinguished cells is an important aspect in many areas of brain research, as the genetic identity contains information about a cells physiological properties and internal state. On top of this, knowledge of the genetic details of each cell can reveal structural organization within tissue. As image-based spatial transcriptomics moves toward applications in tissues with dense cellular packing, accurate assignment of detected mRNA transcripts ("spots") to correct segmented cells becomes increasingly difficult, rendering simple methods insufficient with many incorrect assignments to neighboring cells. Here we introduce SpotDMix, a statistical model for assigning spots to cells by modeling spots as coming from a mixture model of distributions matching segmented cell shapes, with assignment probabilities and shape parameters optimized using the Expectation Maximization algorithm. Performance is assessed and compared against several simple methods in various scenarios on both surrogate data and larval zebrafish data. In all tested scenarios SpotDMix outperforms the simple methods on all evaluated metrics, including individual transcript assignment accuracy, total assigned number of spots per cell error and cell type classification. Further, SpotDMix produces a higher degree of exclusivity between genes which are known to not or rarely co-express.