Weak supervision of H&E slides reveals systems-level biology and functional states that govern therapeutic resistance

Precision oncology lacks scalable tools to assess, at the patient level, systems-level tumor microenvironment (TME) programs driving therapeutic resistance. To address this gap, we trained a weakly-supervised deep learning model that uses routine H&E whole-slide images (WSIs) to derive quantitative activity for therapeutically-relevant TME phenotypes, spanning immune, metabolic, and tumor cell-intrinsic programs. Using 3111 breast cancer H&E WSIs with matched bulk transcriptomics, our model accurately infers these biological states, defined by pathway enrichment scores (AUROC>0.80; PCC>0.64). Validation spanned three levels: (i) tissue-matched multiplexed immunofluorescence, showing concordance between inferred functional states and immune cell fractions (p=0.006-0.106), (ii) blinded reader assessments, confirming localization of phenotype-specific morphology (p<3x10-5), and (iii) multi-institutional patient cohorts, where model-derived phenotypes stratified for clinical response (p<0.045). Unlike methods requiring resource-intensive spatial profiling data for training, our approach leverages widely-available therapeutic outcomes or bulk profiling as slide-level labels to assess functional biology. This strategy offers a scalable complement to spatial Omics for investigating therapeutic resistance across the pan-cancer landscape through using WSIs and clinical outcomes from massive legacy biobanks.