Back

ChatGPT as a medical doctor? A diagnostic accuracy study on common and rare diseases

Mehnen, L.; Gruarin, S.; Vasileva, M.; Knapp, B.

2023-04-26 health informatics
10.1101/2023.04.20.23288859 medRxiv
Show abstract

Seeking medical advice online has become popular in the recent past. Therefore a growing number of people might ask the recently hyped ChatGPT for medical information regarding their conditions, symptoms and differential diagnosis. In this paper we tested ChatGPT for its diagnostic accuracy on a total of 50 clinical case vignettes including 10 rare case presentations. We found that ChatGPT 4 solves all common cases within 2 suggested diagnoses. For rare disease conditions ChatGPT 4 needs 8 or more suggestions to solve 90% of all cases. The performance of ChatGPT 3.5 is consistently lower than the performance of ChatGPT 4. We also compared the performance between ChatGPT and human medical doctors. We conclude that ChatGPT might be a good tool to assist human medical doctors in diagnosing difficult cases, but despite the good diagnostic accuracy, ChatGPT should be used with caution by non-professionals.

Matching journals

The top 8 journals account for 50% of the predicted probability mass.

1
BMJ Health & Care Informatics
● 15 papers in training set
Top 0.1%
9.4%
Similar papers in this journal
2
PLOS Digital Health
● 106 papers in training set
Top 0.7%
9.4%
Similar papers in this journal
3
PLOS ONE
● 5266 papers in training set
Top 22%
7.6%
Similar papers in this journal
4
BMC Medical Informatics and Decision Making
○ 43 papers in training set
Top 0.2%
7.0%
Similar papers in this journal
5
Journal of Medical Internet Research
◐ 87 papers in training set
Top 0.4%
5.3%
Similar papers in this journal
6
JMIR Medical Informatics
◐ 18 papers in training set
Top 0.1%
5.3%
7
JAMIA Open
◐ 42 papers in training set
Top 0.4%
5.3%
8
JMIR Formative Research
◐ 33 papers in training set
Top 0.3%
5.0%
50% of probability mass above
9
Scientific Reports
○ 3612 papers in training set
Top 23%
4.2%
10
International Journal of Medical Informatics
○ 26 papers in training set
Top 0.3%
3.3%
11
Artificial Intelligence in Medicine
○ 17 papers in training set
Top 0.2%
3.1%
12
Journal of the American Medical Informatics Association
◐ 71 papers in training set
Top 1%
3.1%
13
Journal of Biomedical Informatics
○ 47 papers in training set
Top 0.6%
2.6%
14
Journal of Personalized Medicine
○ 28 papers in training set
Top 0.3%
2.1%
15
BMC Medical Education
○ 21 papers in training set
Top 0.3%
2.1%
16
Life
○ 29 papers in training set
Top 0.2%
1.8%
17
Computers in Biology and Medicine
○ 128 papers in training set
Top 3%
1.4%
18
Frontiers in Digital Health
○ 24 papers in training set
Top 0.7%
1.4%
19
JMIRx Med
◐ 32 papers in training set
Top 2%
1.1%
20
Orphanet Journal of Rare Diseases
○ 21 papers in training set
Top 0.4%
1.1%
21
DIGITAL HEALTH
○ 17 papers in training set
Top 0.6%
1.0%
22
Frontiers in Artificial Intelligence
○ 20 papers in training set
Top 0.8%
0.8%
23
Database
◐ 61 papers in training set
Top 1.0%
0.8%
24
Journal of Clinical Medicine
○ 97 papers in training set
Top 4%
0.8%
25
European Journal of Human Genetics
○ 58 papers in training set
Top 1%
0.8%
26
Biology Methods and Protocols
◐ 61 papers in training set
Top 3%
0.6%
27
Frontiers in Psychiatry
○ 87 papers in training set
Top 3%
0.6%
28
Informatics in Medicine Unlocked
○ 22 papers in training set
Top 1%
0.6%

"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.