SalienceNet: an unsupervised Image-to-Image translation method for nuclei saliency enhancement in microscopy images

Automatic segmentation of nuclei in low-light microscopy images remains a difficult task, especially for high-throughput experiments where need for automation is strong. Low saliency of nuclei with respect to the background, variability of their intensity together with low signal-to-noise ratio in these images constitute a major challenge for mainstream algorithms of nuclei segmentation. In this work we introduce SalienceNet, an unsupervised deep learning-based method that uses the style transfer properties of cycleGAN to transform low saliency images into high saliency images, thus enabling accurate segmentation by downstream analysis methods, and that without need for any parameter tuning. We have acquired a novel dataset of organoid images with soSPIM, a microscopy technique that enables the acquisition of images in low-light conditions. Our experiments show that SalienceNet increased the saliency of these images up to the desired level. Moreover, we evaluated the impact of SalienceNet on segmentation for both Otsu thresholding and StarDist and have shown that enhancing nuclei with SalienceNet improved segmentation results using Otsu thresholding by 30% and using StarDist by 26% in terms of IOU when compared to segmentation of non-enhanced images. Together these results show that SalienceNet can be used as a common preprocessing step to automate nuclei segmentation pipelines for low-light microscopy images.