Decision Confidence and Outcome Variability Optimally Regulate Separate Aspects of Hyperparameter Setting

Reinforcement learning models describe how agents learn about the world (value learning), and how they interact with their environment based on the learned information (decision policy). As in any optimization problem, it is important to set the process hyperparameters, a process which also is thought to be learned (meta-learning). Here, we test a key prediction of meta-learning frameworks, namely that there exist one or more meta-signals that govern hyperparameter setting. Specifically, we test whether decision confidence, in a context of varying outcome variability, informs hyperparameter setting. Participants performed a 2-armed bandit task with confidence ratings. Model comparison shows that confidence and outcome variability are differentially involved in hyperparameter setting. A high level of confidence in the previous choice decreased hyperparameter setting of decision noise on the current trial: when a trial was made with low confidence, the choice on the next trial tended to be more explorative (i.e. high decision noise). Outcome variability influenced another hyperparameter, the learning rate for positive prediction errors (thus affecting value learning). Both strategies are rational approaches that maximize earnings at different temporal loci: the modulation by confidence causes more frequent exploration early after a change point, the modulation by outcome variability is advantageous late after a change point. Finally, we show that (reported) confidence in value-based choices reflects the action value of the chosen option (irrespective of the unchosen value). In sum, decision confidence and outcome variability reflect distinct signals that optimally guide the setting of hyperparameters in decision policy and value learning, respectively.