Benchmarking LLM-based Information Extraction Tools for Medical Documents

MotivationMedical documents are a crucial resource for medical research around the world. While troves of valuable health data exist, they are largely computationally inaccessible as hard copies of unstructured text. Moreover, the persistent prevalence of fax machines in medical settings contributes to further degradation of document quality. Digitization of these resources through manual data extraction is time-consuming and resource intensive. However, large language models (LLMs) have recently shown great promise for automated digitization and information extraction (IE), greatly improving upon previous tools in terms of speed and accuracy. ResultsWe reviewed recent LLM-based tools for named entity recognition (NER) and IE from the literature and assessed them with respect to their suitability for use in a clinical setting. We found only two of these tools to be usable out of the box and compared them to LLM foundation models prompted to perform extractions. Using 1000 mock medical documents with paired reference data, we evaluated the tools performance in different scenarios, comparing zero-shot and one-shot prompts as well as unimodal and multimodal (image and text) inputs where possible. The most effective model was OpenAIs GPT 4.1-mini with an average F1 score of 55.6. The best performing local model was Googles Gemma3 with 27B parameters, given image inputs and a zero-shot prompt, with an average F1 score of 41.3. We found the choice of prompting strategy to have minimal impact on extraction performances. We also assessed the effects of image distortions commonly introduced by fax machines and found a significant impact on extraction performance. AvailabilitySource code and data are available on Github at https://github.com/courtotlab/PDF_benchmarking. Supplementary informationSupplementary data are available at Journal Name online.