A Robust Deep Learning Approach for Joint Nuclei Detection and Cell Classification in Pan-Cancer Histology Images

Advanced image processing methods have shown promise in computational pathology, including the extraction of crucial microscopic features from histology images. Accurate detection and classification of cell nuclei from whole-slide images (WSI) play a crucial role in capturing the molecular and morphological landscape of the tissue sample. They enable widespread downstream applications, including cancer diagnosis, prognosis, and discovery of novel markers. Robust nuclei detection and classification are challenging due to the high intra-class variability and inter-class similarity of the microscopic morphological features. This is further compounded by the domain shift arising due to the variability in tissue types, staining protocols, and image acquisition. Motivated by the ability of the recent deep learning techniques to learn complex patterns in a biasfree manner, we develop a novel and robust deep learning model TransNuc, based on vision transformers, for simultaneous detection and classification of cell nuclei from H&E stained WSI. We benchmarked TransNuc on the comprehensive Open Pan-cancer Histology Dataset (PanNuke), sampled from over 20,000 WSI, comprising 19 different tissue types and five clinically important cell classes, namely, Neoplastic, Epithelial, Inflammatory, Connective, and Dead cells. TransNuc exhibited superior performance compared to the state-of-theart, including Hover-Net and Micro-Net. TransNuc was able to learn robust feature representations and thereby perform consistently better for the abundant classes such as neoplastic, and the under-represented classes such as dead cells. Similar performance gains were also obtained for epithelial and connective classes that have a significant inter-class morphological similarity.