Leveraging transformers and large language models with antimicrobial prescribing data to predict sources of infection for electronic health record studies

BackgroundElectronic health records frequently contain extensive unstructured free-text data, but extracting information accurately from these data and at scale is challenging. Using free-text from antibiotic prescribing data as an example, we investigate the performance of modern natural language processing methods (NLP) and large language models (LLMs) as tools for extracting features from medical records. MethodsWe used 938,150 hospital antibiotic prescriptions from Oxfordshire, UK. The 4000 most frequently used free-text indications justifying antibiotic use were labelled by clinical researchers into 11 categories describing the infection source/clinical syndrome being treated and used for model training. Traditional classification methods, fuzzy regex matching and n-grams with XGBoost, were compared against modern transformer models: we fine-tuned generic and domain-specific BERT models, fine-tuned GPT3.5, and investigated few-shot learning with GPT4. Models were evaluated on internal and external test datasets (2000 prescriptions each). Infection sources determined from ICD10 codes were also used for comparisons. ResultsIn internal and external test datasets, the fine-tuned domain-specific Bio+Clinical BERT model averaged an F1 score of 0.97 and 0.98 respectively across the classes and outperformed the traditional regex (F1=0.71 and 0.74) and n-grams/XGBoost (F1=0.86 and 0.84). OpenAIs GPT4 model achieved F1 scores of 0.71 and 0.86 without using labelled training data and a fine-tuned GPT3.5 model F1 scores of 0.95 and 0.97. Comparing infection sources extracted from ICD10 codes to those parsed from free-text indications, free-text indications revealed 31% more specific infection sources. ConclusionModern transformer-based models can efficiently and accurately categorise semi-structured free-text in medical records, such as prescription free-text. Finetuned local transformer models outperform LLMs currently for structured tasks. Few shot LLMs match the performance of traditional NLP without the need for labelling. Transformer-based models have the potential to be used widely throughout medicine to analyse medical records more accurately, facilitating beter research and patient care.