Multimodal-Multitask-Selfsupervised Xdeep-Msi: Explainable Bias-Rejecting Microsatellite Instability Deep Learning System In Colorectal Cancer

AO_SCPLOWBSTRACTC_SCPLOWThe prediction of microsatellite instability (MSI) in colorectal cancer (CRC) using deep learning (DL) techniques directly from hematoxylin and eosin stained slides (H&E) has been shown feasible by independent works. Nonetheless, when available, relevant information from clinical, oncological and family history could be used to further inform DL predictions. The present work analyzes the effects from leveraging multimodal inputs and multitask supervision in a previously published DL system for the prediction of MSI in CRC (xDEEP-MSI). xDEEP-MSI was a multiple bias rejecting DL system based on adversarial networks trained and validated in 1788 patients from a total of 25 participating centers from EPICOLON and HGUA projects. In the present work, xDEEP-MSI is further enriched with weakly supervised learning in multiple molecular alterations (MSI status, K-RAS and BRAF mutations and Lynch Syndrome confirmed by germline mutations), adapted to multimodal inputs with variable degree of completeness (image, age, gender, localization of CRC, revised Bethesda criteria, Amsterdam II criteria and additional oncological history) and a self-supervised multiple instance learning that integrates multiple image-tiles, to obtain patient-level predictions. The AUC, including all three selected tissues (tumor epithelium, mucin and lymphocytic regions) and 5 magnifications, increases from 0.9 {+/-} 0.03, to 0.94 {+/-} 0.02. The sensibility and specificity reaches 92.5% 95%CI(79.6-98.4%) and 93.4% 95%CI(90.0-95.8%) respectively. To the best of our knowledge this is the first work that jointly uses multimodal inputs, multiple instance learning and multiple molecular supervision for the prediction of MSI in CRC from H&E, demonstrating their gains in performance. Prospective validation in an external independent dataset is still required.