Back

Improved Performance of ChatGPT-4 on the OKAP Exam: A Comparative Study with ChatGPT-3.5

Teebagy, S.; Colwell, L.; Wood, E.; Yaghy, A.; Faustina, M.

2023-04-03 ophthalmology
10.1101/2023.04.03.23287957 medRxiv
Show abstract

This study aims to evaluate the performance of ChatGPT-4, an advanced Artificial Intelligence (AI) language model, on the Ophthalmology Knowledge Assessment Program (OKAP) examination compared to its predecessor, ChatGPT-3.5. Both models were tested on 180 OKAP practice questions covering various ophthalmology subject categories. Results showed that ChatGPT-4 significantly outperformed ChatGPT-3.5 (81% vs. 57%; p<0.001), indicating improvements in medical knowledge assessment. The superior performance of ChatGPT-4 suggests potential applicability in ophthalmologic education and clinical decision support systems. Future research should focus on refining AI models, ensuring a balanced representation of fundamental and specialized knowledge, and determining the optimal method of integrating AI into medical education and practice.

Matching journals

The top 7 journals account for 50% of the predicted probability mass.

1
PLOS Digital Health
● 106 papers in training set
Top 0.3%
15.1%
Similar papers in this journal
2
Ophthalmology Science
○ 22 papers in training set
Top 0.1%
6.7%
Similar papers in this journal
3
DIGITAL HEALTH
○ 17 papers in training set
Top 0.1%
6.7%
Similar papers in this journal
4
PLOS ONE
● 5266 papers in training set
Top 24%
6.7%
Similar papers in this journal
5
Scientific Reports
○ 3612 papers in training set
Top 12%
6.3%
Similar papers in this journal
6
British Journal of Ophthalmology
● 14 papers in training set
Top 0.1%
6.3%
7
Computers in Biology and Medicine
○ 128 papers in training set
Top 0.6%
5.5%
50% of probability mass above
8
Journal of Medical Internet Research
◐ 87 papers in training set
Top 0.4%
5.5%
9
F1000Research
○ 88 papers in training set
Top 0.2%
4.9%
10
Eye
○ 11 papers in training set
Top 0.2%
4.3%
11
Translational Vision Science & Technology
● 39 papers in training set
Top 0.2%
4.0%
12
Cureus
○ 68 papers in training set
Top 1%
3.2%
13
BMC Medical Education
○ 21 papers in training set
Top 0.2%
3.2%
14
Bioengineering
○ 29 papers in training set
Top 0.2%
2.4%
15
Annals of Translational Medicine
○ 18 papers in training set
Top 0.2%
1.7%
16
npj Digital Medicine
○ 118 papers in training set
Top 2%
1.7%
17
International Journal of Medical Informatics
○ 26 papers in training set
Top 1%
1.1%
18
Frontiers in Medicine
○ 120 papers in training set
Top 3%
1.0%
19
Neurosurgery
○ 11 papers in training set
Top 0.3%
1.0%
20
Disability and Rehabilitation
○ 11 papers in training set
Top 0.4%
0.8%
21
BMJ Open
● 601 papers in training set
Top 12%
0.8%
22
Journal of Clinical Medicine
○ 97 papers in training set
Top 4%
0.8%
23
Healthcare
○ 17 papers in training set
Top 1%
0.6%
24
Brain Sciences
○ 55 papers in training set
Top 2%
0.6%
25
BMJ Health & Care Informatics
● 15 papers in training set
Top 1%
0.6%

"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.