Optimizing Contingency Management with Reinforcement Learning
Kim, Y.; Brandt, L.; Cheung, K.; Nunes, E. V.; Roll, J.; Luo, S. X.; Liu, Y.
Show abstract
Contingency Management (CM) is a psychological treatment that aims to change behavior with financial incentives. In substance use disorders (SUDs), deployment of CM has been enriched by longstanding discussions around the cost-effectiveness of prized-based and voucher-based approaches. In prize-based CM, participants earn draws to win prizes, including small incentives to reduce costs, and the number of draws escalates depending on the duration of maintenance of abstinence. In voucher-based CM, participants receive a predetermined voucher amount based on specific substance test results. While both types have enhanced treatment outcomes, there is room for improvement in their cost-effectiveness: the voucher-based system requires enduring financial investment; the prize-based system might sacrifice efficacy. Previous work in computational psychiatry of SUDs typically employs frameworks wherein participants make decisions to maximize their expected compensation. In contrast, we developed new frameworks that clinical decision-makers choose actions, CM structures, to reinforce the substance abstinence behavior of participants. We consider the choice of the voucher or prize to be a sequential decision, where there are two pivotal parameters: the prize probability for each draw and the escalation rule determining the number of draws. Recent advancements in Reinforcement Learning, more specifically, in off-policy evaluation, afforded techniques to estimate outcomes for different CM decision scenarios from observed clinical trial data. We searched CM schemas that maximized treatment outcomes with budget constraints. Using this framework, we analyzed data from the Clinical Trials Network to construct unbiased estimators on the effects of new CM schemas. Our results indicated that the optimal CM schema would be to strengthen reinforcement rapidly in the middle of the treatment course. Our estimated optimal CM policy improved treatment outcomes by 32% while maintaining costs. Our methods and results have broad applications in future clinical trial planning and translational investigations on the neurobiological basis of SUDs.
Matching journals
The top 8 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- Reducing maladaptive behavior in neuropsychiatric disorders using network modification 93%
- Network-Based Discovery of Opioid Use Vulnerability in Rats Using the Bayesian Stochastic Block Model 90%
- Synapses, predictions, and prediction errors: a neocortical computational study of MDD using the temporal memory algorithm of HTM. 90%
Similar papers in this journal
- The effect of body image dissatisfaction on goal-directed decision making in a population marked by negative appearance beliefs and disordered eating 92%
- Can alcohol consumption in Germany be reduced by alcohol screening, brief intervention and referral to treatment in primary health care? Results of a simulation study 92%
- Cost-utility of aripiprazole once-monthly versus paliperidone palmitate once-monthly injectable for schizophrenia in China 91%
Similar papers in this journal
- Differential Treatment Benefit Prediction For Treatment Selection in Depression: A Deep Learning Analysis of STAR*D and CO-MED Data 93%
- Computational Mechanisms of Approach-Avoidance Conflict Predictively Differentiate Between Affective and Substance Use Disorders 91%
- The reward-complexity trade-off in schizophrenia 90%
Similar papers in this journal
- Spatial inequities in access to medications for treatment of opioid use disorder highlight scarcity of methadone providers under counterfactual scenarios 92%
- A Bayesian computational model reveals a failure to adapt interoceptive precision estimates across depression, anxiety, eating, and substance use disorders 92%
- Directed exploration in the Iowa Gambling Task: model-free and model-based analyses in a large dataset of young and old healthy participants 92%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.