How Best to Explain Machine Learning Models to Clinicians: A User Study of Explanation Types

Background: Explanations play a crucial role in helping clinicians understand how black-box machine learning models make predictions in clinical settings. Several different types of explanations have been developed, each corresponding to a unique approach for characterizing the relationships between model inputs and predictions. However, it remains unclear what types of explanations are the most valued by clinicians. Objective: To improve the utility of machine learning in clinical settings, we aimed to evaluate how different explanation methods are valued by clinicians across clinically important metrics, such as importance, trust, understanding, and how explanations affect clinicians' thinking about patients. Methods: We conducted a user study of 39 critical care and hospital medicine nurses and physicians to compare attribution, counterfactual, and rule-based explanations. We analyzed the impact of each type of explanation on clinicians' trust in and understanding of the predictions made by machine learning models, how well clinicians understood the explanation, and how the explanation affected what they thought were the most important features for determining patients' status. We also assessed clinicians' preferences for the representation of different types of explanations. Results: Clinicians consider explanations of clinical machine learning models important, with physicians perceiving explanations as more important after interacting with them than nurses. All explanation types affected clinicians across all measured dimensions, with attribution explanations having the most significant positive effects on all measured dimensions. Moreover, nearly half of clinicians preferred viewing multiple explanation types together. Conclusions: It is important to provide explanations for predictions made by machine-learning models in clinical settings. When implementing machine learning explanations in these settings, developers should prioritize attribution explanations while allowing for multiple types of explanations to be shown. Furthermore, the development of new explanation methods should be tailored towards specific clinical roles, as nurses and physicians may utilize explanations differently to support their respective workflows.