Learning to Treat Hypotensive Episodes in Sepsis Patients Using a Counterfactual Reasoning Framework

The optimal treatment strategy for volume resuscitation and vasopressor dosing to combat hypotensive episodes in septic patients remains a subject of ongoing controversy and can vary from clinician to clinician. We develop a machine learning approach to guide a fluid and vasopressor dosing strategy that adapts to patient-specific clinical states to improve the survival of septic patients. We adopt a model-free reinforcement learning (RL) framework in a continuous action space with a clinically significant reward function, and use a Switching Generalized Linear Model (SGLM) to characterize patient-specific clinical states. We use retrospective data from the MIMIC III database to train this model to learn volume resuscitation and vasopressor dosing strategies among the 5,366 patients (totalling 352,328 unique hourly measurements) with ICU-onset sepsis or septic shock, as diagnosed by the Sepsis-3 definition. The RL agent receives short- and long-term rewards associated with optimizing in-hospital survival and avoiding end-organ damage to learn volume resuscitation and vasopressor dosing strategies. On average, the RL agent learns to resuscitate patients earlier than clinicians with a fluid bolus (one hour vs. four hours after the diagnosis of sepsis), and improves the expected survival by {approx} 3%. Our preliminary results indicate that adherence to RL-based individualized fluid and vasopressor dosing recommendations is associated with a significant mortality reduction in septic patients, even after adjusting for severity of illness.