Reinforcement learning optimization of automated insulin delivery in type 1 and type 2 diabetes mellitus

Closed-loop insulin delivery systems have proven effective in regulating blood glucose (BG) concentration, thereby reducing the burden of self-care in the management of type 1 and type 2 diabetes mellitus. However, the prevalence of unexpected disturbances resulting from oral glucose intake represents a considerable challenge to the full automation of these systems. Here, we propose an actor-critic reinforcement learning (RL) framework implemented within environments governed by compartmental ordinary differential equation models of the glucose-insulin-glucagon-incretins dynamics. This approach was employed to optimize automated insulin delivery in virtual patients with type 1 and type 2 diabetes mellitus, under scenarios involving unforeseen BG disturbances. The resulting optimal RL policies were tested in silico on virtual patients subjected to three unannounced glucose disturbances over the course of a day. The findings demonstrated that optimal RL policies could sustain the BG a significantly higher percentage of time within the normoglycemic range and a significantly lower percentage of time below the normoglycemic range in comparison to either continuous or discrete proportional-integral-derivative control algorithms. These results set the basis for developing new approaches to optimizing automated dosing regimens for chronic disease management. Author SummaryManaging diabetes requires constant attention to glucose levels and the corresponding adjustment of insulin doses, which can be demanding for insulin-dependent patients. Semi-automated systems, frequently referred to as "artificial pancreas", aim to mitigate this burden by adjusting insulin delivery based on glucose levels obtained from a continuous glucose sensor. However, these systems underperform in scenarios involving unexpected glucose disturbances, such as those triggered by the omission of meal announcing within the system interface. In the present study, we developed a computer-based learning approach to identify deep network-based functions that determine the appropriate insulin doses to regulate glucose in real time and mitigate unannounced disturbances. This approach has been demonstrated to be effective in the management of Type 1 and Type 2 diabetes and it does not require any additional input other than the continuous glucose measurements. The findings of our study demonstrate that our functions were able to maintain glucose levels within a healthy range for a longer period of time than the standard functions, while also reducing the risk of threating low glucose levels. This research contributes to the development of fully automated insulin delivery systems that are capable of adapting to real-life situations of individuals living with diabetes.