Utilising Large Language Models for the Automated Mapping of Medical Research to Translational Stages
Clapham, M.; Oldmeadow, C.; Deeming, S.; Riveros, C.
Show abstract
Classification of research articles according to translational research stages enables funding bodies, academic and medical institutes, and policymakers to objectively assess the distribution of resources across the research spectrum. We aim to utilise Large Language Models (LLM) to classify medical research papers into translational research levels based on their titles and abstracts, comparing performance across a range of LLMs, multiple runs and a bag of words (BoW) baseline. We quantify the performance of open-weight LLMs against a human-labelled data set of 318 medical research papers. Using a description of translational levels, the LLMs showed good performance with an F1 score of 0.83 ahead of a baseline BoW approach of 0.68. We show that LLMs can accurately classify titles and abstracts into translational levels within a fully automated pipeline.
Matching journals
The top 6 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- Ethical review of clinical research with generative AI: Evaluating ChatGPT’s accuracy and reproducibility 94%
- Diversity and inclusion: A hidden additional benefit of Open Data 94%
- Inferring Gender from First Names: Comparing the Accuracy of Genderize, Gender API, and the gender R Package on Authors of Diverse Nationality 93%
Similar papers in this journal
- Automating literature screening and curation with applications to computational neuroscience 95%
- A Novel Question-Answering Framework for Automated Abstract Screening Using Large Language Models 95%
- Annotation-preserving machine translation of English corpora to validate Dutch clinical concept extraction tools 93%
Similar papers in this journal
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.