Performance of ChatGPT on the MCAT: The Road to Personalized and Equitable Premedical Learning
Bommineni, V. L.; Bhagwagar, S. Z.; Balcarcel, D.; Bommineni, V.; Davatzikos, C.; Boyer, D. L.
Show abstract
Despite an increasingly diverse population, an unmet demand for undergraduates from underrepresented racial and ethnic minority (URM) backgrounds exists in the field of medicine as a result of financial hurdles and insufficient educational support faced by URM students in the premedical journey. With the capacity to provide highly individualized and accessible no- or low-cost dynamic instruction, large language models (LLMs) and their chatbot derivatives are posed to change this dynamic and subsequently help shape a more diverse future physician workforce. While studies have established the passing performance and insightful explanations of one of the most accurate LLM-powered chatbots to date--Chat Generative Pre-trained Transformer (ChatGPT)--on standardized exams such as medical licensing exams, the role of ChatGPT in premedical education remains unknown. We evaluated the performance of ChatGPT on the Medical College Admission Test (MCAT), a standardized 230-question multiple choice exam that assesses a broad range of competencies in the natural, physical, social, and behavioral sciences as well as critical analysis and reasoning. Depending on its visual item response strategy, ChatGPT performed at or above the median performance of 276,779 student test takers on the MCAT. Additionally, ChatGPT-generated answers demonstrated both a high level of agreement with the official answer key as well as insight into its explanations. Based on these promising results, we anticipate two primary applications of ChatGPT and future LLM iterations in premedical education: firstly, such models could provide free or low-cost access to personalized and insightful explanations of MCAT competency-related questions to help students from all socioeconomic and URM backgrounds. Secondly, these models could be used to generate additional test questions by test-makers or for targeted preparation by pre-medical students. These applications of ChatGPT in premedical education could be an invaluable, innovative path forward to increase diversity and improve equity among premedical students.
Matching journals
The top 5 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- A Framework to Assess Clinical Safety and Hallucination Rates of LLMs for Medical Text Summarisation 91%
- From Tool to Teammate: A Randomized Controlled Trial of Clinician-AI Collaborative Workflows for Diagnosis 91%
- Evaluating large language model workflows in clinical decision support: referral, triage, and diagnosis 91%
Similar papers in this journal
- Performance of Generative Pretrained Transformer on the National Medical Licensing Examination in Japan 93%
- Collaborative intelligence in AI: Evaluating the performance of a council of AIs on the USMLE 93%
- Ethical review of clinical research with generative AI: Evaluating ChatGPT’s accuracy and reproducibility 91%
Similar papers in this journal
- Performance of ChatGPT on Chinese National Medical Licensing Examinations: A Five-Year Examination Evaluation Study for Physicians, Pharmacists and Nurses 92%
- Large language models for generating medical examinations: systematic review 91%
- Training Doctoral Students in Critical Thinking and Experimental Design using Problem-based Learning 91%
Similar papers in this journal
- Quality of Human Expert vs. Large Language Model Generated Multiple Choice Questions in the Field of Mechanical Ventilation 92%
- Development and Prospective Validation of a Transparent Deep Learning Algorithm for Predicting Need for Mechanical Ventilation 86%
- Rethinking Blood Eosinophils for Assessing ICS Response in COPD: A Post-Hoc Analysis from FLAME 81%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.