Capability of GPT-4V(ision) in Japanese National Medical Licensing Examination
Nakao, T.; Miki, S.; Nakamura, Y.; Kikuchi, T.; Nomura, Y.; Hanaoka, S.; Yoshikawa, T.; Abe, O.
Show abstract
BackgroundPrevious research applying large language models (LLMs) to medicine was focused on text-based information. Recently, multimodal variants of LLMs acquired the capability of recognizing images. ObjectiveTo evaluate the capability of GPT-4V, a recent multimodal LLM developed by OpenAI, in recognizing images in the medical field by testing its capability to answer questions in the 117th Japanese National Medical Licensing Examination. MethodsWe focused on 108 questions that had one or more images as part of a question and presented GPT-4V with the same questions under two conditions: 1) with both the question text and associated image(s), and 2) with the question text only. We then compared the difference in accuracy between the two conditions using the exact McNemars test. ResultsAmong the 108 questions with images, GPT-4Vs accuracy was 68% when presented with images and 72% when presented without images (P = .36). ConclusionsThe additional information from the images did not significantly improve the performance of GPT-4V in the Japanese Medical Licensing Examination.
Matching journals
The top 5 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- Performance of Generative Pretrained Transformer on the National Medical Licensing Examination in Japan 98%
- Ethical review of clinical research with generative AI: Evaluating ChatGPT’s accuracy and reproducibility 94%
- Cardiology Knowledge Assessment of Retrieval-Augmented Open versus Proprietary Large Language Models 93%
Similar papers in this journal
- One LLM is not Enough: Harnessing the Power of Ensemble Learning for Medical Question Answering 94%
- Design and implementation of a system for automated monitoring of adherence to evidenced-based clinical guideline recommendations 92%
- Improving Patient Engagement in Phase 2 Clinical Trials with a Trial-specific Patient Decision Aid (tPDA): A Development and Usability Study 91%
Similar papers in this journal
- Performance of ChatGPT on Chinese National Medical Licensing Examinations: A Five-Year Examination Evaluation Study for Physicians, Pharmacists and Nurses 96%
- Large language models for generating medical examinations: systematic review 95%
- Medical students' perceptions towards artificial intelligence in education and practice: A multinational, multicenter cross-sectional study 91%
Similar papers in this journal
- A Deep Learning Based Smartphone Application for Early Detection of Nasopharyngeal Carcinoma Using Endoscopic Images 94%
- Evaluating large language model workflows in clinical decision support: referral, triage, and diagnosis 91%
- A human-in-the-loop explanation framework for morphologically transparent AI predictions from whole-slide images 91%
Similar papers in this journal
- ai-corona : Radiologist-Assistant Deep Learning Framework for COVID-19 Diagnosis in Chest CT Scans 93%
- PathEX: Make Good Choice for Whole Slide Image Extraction 93%
- Influence of open-source virtual-reality based gaze training on navigation performance in Retinitis pigmentosa patients in a crossover randomized controlled trial 92%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.