Back

Standardizing and Scaffolding Healthcare AI-Chatbot Evaluation

Hua, Y.; Xia, W.; Bates, D. W.; Hartstein, L.; Kim, H. T.; Li, M. L.; Nelson, B. W.; Stromeyer, C.; King, D.; Suh, J.; Zhou, L.; Torous, J.

2024-07-21 health policy
10.1101/2024.07.21.24310774 medRxiv
Show abstract

The rapid rise of healthcare chatbots, valued at $787.1 million in 2022 and projected to grow at 23.9% annually through 2030, underscores the need for robust evaluation frameworks. Despite their potential, the absence of standardized evaluation criteria and rapid AI advancements complicate assessments. This study addresses these challenges by developing the first comprehensive evaluation framework inspired by health app regulations and integrating insights from diverse stakeholders. Following PRISMA guidelines, we reviewed 11 existing frameworks, refining 271 questions into a structured framework encompassing three priority constructs, 18 second-level constructs, and 60 third-level constructs. Our framework emphasizes safety, privacy, trustworthiness, and usefulness, aligning with recent concerns about AI in healthcare. This adaptable framework aims to serve as the initial step in facilitating the responsible integration of chatbots into healthcare settings.

Matching journals

The top 5 journals account for 50% of the predicted probability mass.

1
Journal of the American Medical Informatics Association
◐ 71 papers in training set
Top 0.2%
15.2%
Similar papers in this journal
2
PLOS ONE
● 5266 papers in training set
Top 16%
11.1%
Similar papers in this journal
3
Journal of Medical Internet Research
◐ 87 papers in training set
Top 0.1%
11.1%
Similar papers in this journal
4
BMJ Health & Care Informatics
● 15 papers in training set
Top 0.1%
9.8%
Similar papers in this journal
5
npj Digital Medicine
○ 118 papers in training set
Top 0.7%
7.9%
Similar papers in this journal
50% of probability mass above
6
PLOS Digital Health
● 106 papers in training set
Top 1%
5.2%
7
BMJ Open
● 601 papers in training set
Top 5%
4.3%
8
BMC Medical Informatics and Decision Making
○ 43 papers in training set
Top 0.7%
2.7%
9
Scientific Reports
○ 3612 papers in training set
Top 42%
2.4%
10
Medical Decision Making
○ 12 papers in training set
Top 0.1%
2.1%
11
DIGITAL HEALTH
○ 17 papers in training set
Top 0.4%
1.7%
12
BMJ Open Quality
● 17 papers in training set
Top 0.3%
1.7%
13
Healthcare
○ 17 papers in training set
Top 0.3%
1.5%
14
Health Policy
○ 11 papers in training set
Top 0.4%
1.3%
15
Frontiers in Public Health
○ 148 papers in training set
Top 5%
1.1%
16
JMIR Medical Informatics
◐ 18 papers in training set
Top 0.6%
1.1%
17
JMIR Research Protocols
◐ 21 papers in training set
Top 0.9%
1.0%
18
JAMA Network Open
● 130 papers in training set
Top 3%
1.0%
19
Clinical Trials
○ 11 papers in training set
Top 0.2%
0.9%
20
Frontiers in Digital Health
○ 24 papers in training set
Top 1%
0.9%
21
JMIRx Med
◐ 32 papers in training set
Top 2%
0.8%
22
BMC Public Health
○ 158 papers in training set
Top 5%
0.8%
23
Communications Medicine
○ 113 papers in training set
Top 4%
0.8%
24
Cancer Medicine
○ 26 papers in training set
Top 1%
0.6%
25
Journal of Biomedical Informatics
○ 47 papers in training set
Top 1%
0.6%
26
Trials
○ 29 papers in training set
Top 1%
0.6%
27
BMC Medical Education
○ 21 papers in training set
Top 0.7%
0.6%
28
F1000Research
○ 88 papers in training set
Top 4%
0.6%

"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.