Can large language models extract travel history from clinical text? A multi-annotator benchmark study
Kanithi, P.; Pradhan, A.; Cui, H.; Johnstone, K.; Luk, A.; Hammond, R.; Brown, C.; Walker, A. S.; Eyre, D. W.
Show abstract
International travel can increase infection risk and spread antimicrobial resistance (AMR), yet clinical systems rarely capture travel history in a structured format. Extracting this information from free-text clinical documentation at scale could improve alerts for high-consequence infections and AMR surveillance. We developed an annotation framework covering key aspects of travel history, including destination, exposures, travel duration, and multiple temporal variables. The framework was used to annotate a corpus of 100 clinical case reports by five clinicians, yielding over 5,000 clinician annotations. Several LLMs were evaluated on the same extraction task, with their outputs compared against clinician consensus. Agreement among clinicians was high for straightforward variables such as presence of travel and destination, but dropped substantially for temporal variables such as time since return and symptom onset relative to return. LLMs showed a similar pattern. They performed well on high-level variables but struggled with the same temporal fields that challenged clinicians. Model errors were also concentrated in cases where clinicians disagreed. Across models, outputs were more sensitive to sampling temperature than to prompt wording. A smaller open-weight model performed similarly to substantially larger models on several variables but remained weaker on temporal fields. Overall, LLMs could extract some travel history variables, but reliable extraction of temporal information remained difficult.
Matching journals
The top 13 journals account for 50% of the predicted probability mass.
Similar papers in this journal
Similar papers in this journal
- Evaluating and Mitigating Limitations of Large Language Models in Clinical Decision Making 92%
- A framework for identifying regional outbreak and spread of COVID-19 from one-minute population-wide surveys 89%
- Direct Antimicrobial Resistance Prediction from clinical MALDI-TOF mass spectra using Machine Learning 89%
Similar papers in this journal
- Convolutional neural networks quantify antibiotic resistance in Mycobacterium tuberculosis with diagnostic grade accuracy and predict treatment response 91%
- Integrating a host transcriptomic biomarker with a large language model for diagnosis of lower respiratory tract infection 90%
- Deep representation learning for clustering longitudinal survival data from electronic health records 89%
Similar papers in this journal
- Rapid implementation of real-time SARS-CoV-2 sequencing to investigate healthcare-associated COVID-19 infections 90%
- An interactive COVID-19 virus Mutation Tracker (CovMT) with a particular focus on critical mutations in the Receptor Binding Domain (RBD) region of the Spike protein 89%
- Effect of a community-based behavioural intervention bundle to improve antibiotic use and patient management in Burkina Faso and DR Congo: a cluster randomized controlled trial 87%
Similar papers in this journal
- Large Language Models for Zero-Shot Procedure Extraction in Orthopedic Surgery: A Comparative Evaluation 91%
- ProtAlign-ARG: Antibiotic Resistance Gene Characterization Integrating Protein Language Models and Alignment-Based Scoring 91%
- Toward Trustworthy Chatbots: A Protocol for Red Teaming for Health Related Conversations 91%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.