Impact of a machine learning-powered algorithm on pathologist HER2 IHC scoring in breast cancer
Shamshoian, J.; Shanis, Z.; Cabeen, R.; Yu, L.; Chakraborty, S.; Thibault, M.; Martin, B.; Padigela, H.; Juyal, D.; Javed, S. A.; Qian, W.; Kim, J.; Gerardin, Y.; Rucker, B.; Brosnan-Cashman, J.; Pokkalla, H.; Mehta, J.; Taylor-Weiner, A.; Walk, E.; Beck, A.; Montalto, M. C.; Glass, B.; Balasubramanian, S.
Show abstract
BackgroundHER2 expression level is a key factor in determining the optimal treatment course for breast cancer patients. Roughly 15% of breast cancers are HER2(+), and determination of HER2 status is routinely assessed by immunohistochemistry (IHC). Accurate assessment of the HER2 IHC score by pathologists is therefore critical, especially in light of novel therapeutic approaches demonstrating efficacy in the HER2-low setting. However, there is an opportunity to improve inter-pathologist agreement at the lower levels of HER2 scoring (0, 1+, and 2+). MethodsA machine learning model (AIM-HER2) was developed to generate accurate, slide-level HER2 scores aligned with ASCO-CAP guidelines in clinical breast cancer HER2 IHC specimens. AIM-HER2 was assessed as an AI-assist tool in a retrospective reader study, where 20 HER2-trained pathologists scored breast cancer cases (N=200) with and without AIM-HER2 assistance using a 2-cohort crossover design with a 3-week washout. A separate panel of 5 expert HER2 pathologists read all 200 cases manually to establish reference scores. ResultsIn a significant fraction of cases examined, less than 70% inter-pathologist agreement was observed. When used as an AI assist tool, AIM-HER2 improved inter-rater agreement overall and specifically at the 0/1+ and 1+/2+ cutoffs. Similarly, AIM-HER2 AI-assist significantly increased PPA at the 0/1+ and 1+/2+ cutoffs. When interacting with the AI-assist tool, pathologists displayed a wide range of override rates, and the quality of a pathologists overrides was correlated with their manual accuracy. Lastly, the impact of the reference panel on AIM-HER2 accuracy metrics was assessed, revealing that measurements of model accuracy are highly dependent on reference panel composition. ConclusionsThe use of AIM-HER2 as an AI-assist tool for scoring HER2 IHC in breast cancer may improve pathologist reproducibility and accuracy, particularly at the 0/1+ and 1+/2+ cutoffs.
Matching journals
The top 6 journals account for 50% of the predicted probability mass.