A deep learning and graph-based approach to characterise the immunological landscape and spatial architecture of colon cancer tissue

Tumour immunity is key for the prognosis and treatment of colon adenocarcinoma, but its characterisation remains cumbersome and expensive, requiring sequencing or other complex assays. Detecting tumour-infiltrating lymphocytes in haematoxylin and eosin (H&E) slides of cancer tissue would provide a cost-effective alternative to support clinicians in treatment decisions, but inter- and intra-observer variability can arise even amongst experienced pathologists. Furthermore, the compounded effect of other cells in the tumour microenvironment is challenging to quantify but could yield useful additional biomarkers. We combined RNA sequencing, digital pathology and deep learning through the InceptionV3 architecture to develop a fully automated computer vision model that detects prognostic tumour immunity levels in H&E slides of colon adenocarcinoma with an area under the curve (AUC) of 82%. Amongst tumour infiltrating T cell subsets, we demonstrate that CD8+ effector memory T cell patterns are most recognisable algorithmically with an average AUC of 83%. We subsequently applied nuclear segmentation and classification via HoVer-Net to derive complex cell-cell interaction graphs, which we queried efficiently through a bespoke Neo4J graph database. This uncovered stromal barriers and lymphocyte triplets that could act as structural hallmarks of low immunity tumours with poor prognosis. Our integrated deep learning and graph-based workflow provides evidence for the feasibility of automated detection of complex immune cytotoxicity patterns within H&E-stained colon cancer slides, which could inform new cellular biomarkers and support treatment management of this disease in the future.