Performance of Contemporary Large Language Models on the National Prosthodontic Resident Examination: A Multi-Year Comparative Study
Chen, Y.-w.; Bhunyanaphakul, V.; Kosago, P.; Finkelman, M.; Lee, D. J.
Show abstract
Purpose: To compare the performance of three large language models (LLMs), ChatGPT-o1, Grok 3, and Gemini 2.0 Pro, on the American College of Prosthodontists' National Prosthodontic Resident Examination (NPRE). Materials and Methods: Official NPRE examinations from years 2015-2024 were retrieved from the American College of Prosthodontists website and used to prompt each LLM. Paid access tiers and advanced reasoning were enabled. Responses from LLMs were recorded and compared against the official answer keys for accuracy, variability across years, and consistency of responses. Duplicate questions were retained for the primary analysis and removed in a sensitivity analysis. Accuracy (percent correct) was compared using generalized estimating equations binary logistic regression (primary); Cochran's Q with Bonferroni-adjusted McNemar tests (sensitivity). Year-to-year variability (SD of annual accuracy) was compared using weighted Levene tests. Consistency across duplicate items was quantified as the percentage of perfect matches and compared using Cochran's Q with Bonferroni-adjusted McNemar tests. Results: Across 1,487 questions, overall accuracy differed among models (global P < 0.001). ChatGPT-o1 achieved 78.4% correct, Grok 3 76.4%, and Gemini 2.0 Pro 73.2%; ChatGPT-o1 and Grok 3's accuracy significantly exceeded Gemini 2.0 Pro (both P [≤] 0.016), with no significant difference between ChatGPT-o1 and Grok 3 (P = 0.099). After removing duplicates (n = 1,200), results were similar: ChatGPT-o1, Grok 3, and Gemini 2.0 Pro were 77.1%, 76.6%, and 71.5% correct, respectively; ChatGPT-o1 and Grok 3's accuracy was significantly higher than Gemini 2.0 Pro (both P < 0.001). Variability across years did not differ significantly (P [≥] 0.097). Grok 3 showed significantly higher perfect-match consistency (79.2%) than ChatGPT-o1 (69.4%) and Gemini 2.0 Pro (64.5%) (P [≤] 0.010). Conclusions: Contemporary LLMs showed 70-80% accuracy on postgraduate prosthodontic multiple-choice content. While promising for educational support, residual error and inconsistency warrant human oversight and transparent methods reporting before curricular adoption.
Matching journals
The top 4 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- COVID-19 challenges to dentistry in the new pandemic epicenter: Brazil 93%
- What are the impacts of oral complications from cancer therapy on the quality of life of children? A protocol to update a scoping review 91%
- Effect of Maxillary Expansion and Protraction on the Oropharyngeal Airway in Individuals with Non-syndromic Cleft Palate with or without Cleft Lip 91%
Similar papers in this journal
- Silicone toothbrushes: A scoping review of an underutilized tool in global oral health 91%
- Proposed standards for prosthetic foot reuse and considerations for donation of used prosthetic feet to low-and middle-income countries 89%
- A Scoping Review of Deaf Awareness Programs in Health Professional Education 89%
Similar papers in this journal
- Use of assistive technology to assess distal motor function in subjects with neuromuscular disease 89%
- Ethical review of clinical research with generative AI: Evaluating ChatGPT’s accuracy and reproducibility 89%
- Inferring Gender from First Names: Comparing the Accuracy of Genderize, Gender API, and the gender R Package on Authors of Diverse Nationality 88%
Similar papers in this journal
- Knowledge and aptitude of early childhood, primary and/or secondary education teachers referred to first aid measures in dental trauma in the province of Seville (Spain.) 91%
- Language comprehension developmental milestones in typically developing children assessed by the new Language Phenotype Assessment (LPA) 84%
- Validity and wear compliance of wrist-wearable activity trackers in free-living schoolchildren: validation study 83%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.