From text to tables: Zero-shot extraction of structured clinical data from free-text CT scan reports using foundational large language models
Hongslo, A.; Gupta, A.; Nguyen, Q.; Caldwell, J.; Choi, B.; Harvey, C. J.; Thompson, J.; Mazzotti, D.; Yao, Z.; Noheria, A.
Show abstract
BackgroundLarge language models (LLMs) are being explored for multiple applications in medical research, including medical text classification. We evaluate the performance of 5 off-the-shelf LLMs for classifying free-text CT angiography reports for pulmonary embolism (PE)- related diagnostic labels. MethodsWe assessed 1,025 manually labeled CT reports using 5 LLMs (ChatGPT-4o, Llama 3.3 70b, Llama 3.1 8b, Llama 3.2 3b, Mixtral 8x7b) with zero-shot prefix prompts. Labels included acute PE, bilateral PE, and large PE. Voting ensemble models combining multiple LLM outputs were also tested. ResultsLlama 3.3 70b and ChatGPT-4o outperformed smaller models for all classification tasks. Highest accuracies were 96.6% (acute PE), 92.7% (bilateral PE), and 82.6% (large PE). Voting ensemble models offered no or minimal improvement in classification performance. ConclusionsOff-the-shelf LLMs, particularly larger ones, can classify free-text reports with high accuracy using simple prompts. Further work is needed to optimize prompting strategies and evaluate hybrid approaches.
Matching journals
The top 3 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- Designing a computer-assisted diagnosis system for cardiomegaly detection and radiology report generation 95%
- Cardiology Knowledge Assessment of Retrieval-Augmented Open versus Proprietary Large Language Models 93%
- Predictability and Stability Testing to Assess Clinical Decision Instrument Performance for Children After Blunt Torso Trauma 93%
Similar papers in this journal
- A Comparative Analysis of Privacy-Preserving Large Language Models For Automated Echocardiography Report Analysis 95%
- Automated stratification of trauma injury severity across multiple body regions using multi-modal, multi-class machine learning models 94%
- Use of unstructured text in prognostic clinical prediction models: a systematic review 93%
Similar papers in this journal
- EchoGraph System for Automated Quality Assessment of Echocardiography Reports 94%
- From Tool to Teammate: A Randomized Controlled Trial of Clinician-AI Collaborative Workflows for Diagnosis 93%
- Evaluating large language model workflows in clinical decision support: referral, triage, and diagnosis 93%
Similar papers in this journal
- ARDSFlag: An NLP/Machine Learning Algorithm to Visualize and Detect High-Probability ARDS Admissions Independent of Provider Recognition and Billing Codes 94%
- Optimized Feature Selection and Advanced Machine Learning for Stroke Risk Prediction in Revascularized Coronary Artery Disease Patients 93%
- OASIS+: leveraging machine learning to improve the prognostic accuracy of OASIS severity score for predicting in-hospital mortality 92%
Similar papers in this journal
- A method for rapid machine learning development for data mining with Doctor-In-The-Loop 94%
- ChatGPT Provides Inconsistent Risk-Stratification of Patients With Atraumatic Chest Pain 94%
- Leveraging Machine Learning for Enhanced and Interpretable Risk Prediction of Venous Thromboembolism in Acute Ischemic Stroke Care 93%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.