Automated abstraction of clinical parameters of multiple myeloma from real-world clinical notes using large language models

BackgroundReal-world evidence (RWE) is increasingly recognized as a valuable type of oncology research but extracting fit-for-purpose real-world data (RWD) from electronic health records (EHRs) remains challenging. Manual abstraction from free-text clinical documents, although the gold standard for information extraction, is resource-intensive. RWD generation using natural language processing (NLP) has been limited by performance ceilings and annotation requirements, which recent LLMs improve on. We evaluate new NLP workflows in abstracting multiple myeloma (MM) information from de-identified EHRs. MethodsNLP workflows (BERT and Llama-based using various prompt types) were developed for 12 MM-specific data fields and evaluated with manually curated data from 125 clinical notes. The best Llama-based workflow for three data fields was applied to all recent notes in selected charts to generate patient journey timelines. ResultsAverage F1 for the best Llama and BERT workflows was 0.82 and 0.65 respectively. Best workflow performance ranged across the data fields (F1 = 0.59-0.99). Statistical analysis of the results showed model size, inter-rater reliability (IRR), variable type, and prompt design significantly predicted workflow performance, in descending order of significance (p < 0.05). ConclusionThe overall performance improvements seen with larger LLMs and chain-of-thought prompting was greater in ambiguous data fields. IRR can be used to prioritize NLP resources and increase efficiency of RWD generation without sacrificing data quality.