Assessing the Impact of Pretraining Domain Relevance on Large Language Models Across Various Pathology Reporting Tasks

Deep learning (DL) algorithms continue to develop at a rapid pace, providing researchers access to a set of tools capable of solving a wide array of biomedical challenges. While this progress is promising, it also leads to confusion regarding task-specific model choices, where deeper investigation is necessary to determine the optimal model configuration. Natural language processing (NLP) has the unique ability to accurately and efficiently capture a patients narrative, which can improve the operational efficiency of modern pathology laboratories through advanced computational solutions that can facilitate rapid access to and reporting of histological and molecular findings. In this study, we use pathology reports from a large academic medical system to assess the generalizability and potential real-world applicability of various deep learning-based NLP models on reports with highly specialized vocabulary and complex reporting structures. The performance of each NLP model examined was compared across four distinct tasks: 1) current procedural terminology (CPT) code classification, 2) pathologist classification, 3) report sign-out time regression, and 4) report text generation, under the hypothesis that models initialized on domain-relevant medical text would perform better than models not attuned to this prior knowledge. Our study highlights that the performance of deep learning-based NLP models can vary meaningfully across pathology-related tasks. Models pretrained on medical data outperform other models where medical domain knowledge is crucial, e.g., current procedural terminology (CPT) code classification. However, where interpretation is more subjective (i.e., teasing apart pathologist-specific lexicon and variable sign-out times), models with medical pretraining do not consistently outperform the other approaches. Instead, fine-tuning models pretrained on general or unrelated text sources achieved comparable or better results. Overall, our findings underscore the importance of considering the nature of the task at hand when selecting a pretraining strategy for NLP models in pathology. The optimal approach may vary depending on the specific requirements and nuances of the task, and related text sources can offer valuable insights and improve performance in certain cases, contradicting established notions about domain adaptation. This research contributes to our understanding of pretraining strategies for large language models and further informs the development and deployment of these models in pathology-related applications.