Back

Evaluating Risk Progression in Mental Health Chatbots Using Escalating Prompts

Heston, T. F.

2023-09-12 psychiatry and clinical psychology
10.1101/2023.09.10.23295321 medRxiv
Show abstract

The safety of large language models (LLMs) as mental health chatbots is not fully established. This study evaluated the risk escalation responses of publicly available ChatGPT conversational agents when presented with prompts of increasing depression severity and suicidality. The average referral point to a human was at the midpoint of escalating prompts. However, most agents only definitively recommended professional help at the highest level of risk. Few agents included crisis resources like suicide hotlines. The results suggest current LLMs may fail to escalate mental health risk scenarios appropriately. More rigorous testing and oversight are needed before deployment in mental healthcare settings.

Matching journals

The top 4 journals account for 50% of the predicted probability mass.

1
Frontiers in Psychiatry
○ 87 papers in training set
Top 0.1%
22.3%
Similar papers in this journal
2
PLOS ONE
● 5266 papers in training set
Top 14%
12.9%
Similar papers in this journal
3
Frontiers in Digital Health
○ 24 papers in training set
Top 0.1%
12.1%
Similar papers in this journal
4
Journal of Medical Internet Research
◐ 87 papers in training set
Top 0.1%
9.8%
Similar papers in this journal
50% of probability mass above
5
JMIR Formative Research
◐ 33 papers in training set
Top 0.1%
9.8%
Similar papers in this journal
6
npj Digital Medicine
○ 118 papers in training set
Top 1%
3.3%
7
JMIR Research Protocols
◐ 21 papers in training set
Top 0.4%
2.4%
8
PLOS Digital Health
● 106 papers in training set
Top 2%
2.4%
9
BJPsych Open
● 29 papers in training set
Top 0.4%
1.5%
10
JAMIA Open
◐ 42 papers in training set
Top 1%
1.5%
11
BMC Health Services Research
○ 51 papers in training set
Top 1%
1.4%
12
JMIRx Med
◐ 32 papers in training set
Top 1%
1.1%
13
BMJ Open
● 601 papers in training set
Top 11%
1.1%
14
Journal of Affective Disorders
○ 92 papers in training set
Top 1%
0.9%
15
DIGITAL HEALTH
○ 17 papers in training set
Top 0.7%
0.9%
16
Psychiatry Research
○ 41 papers in training set
Top 0.9%
0.9%
17
Scientific Reports
○ 3612 papers in training set
Top 71%
0.9%
18
International Journal of Environmental Research and Public Health
○ 128 papers in training set
Top 5%
0.9%
19
Nature Medicine
○ 125 papers in training set
Top 4%
0.6%
20
BMC Psychiatry
○ 25 papers in training set
Top 0.8%
0.6%

"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.