Implementation, Usability, and Workflow Integration of an ML-based CDS Tool: A Qualitative Evaluation of User Requirements

BackgroundPersonalized asthma management depends on a clinicians ability to efficiently review patients data and make timely clinical decisions. Unfortunately, efficient and effective review of these data is impeded by the varied format, location, and workflow of data acquisition, storage, and processing in the electronic health record. While machine learning and clinical decision support tools are well-positioned as potential solutions, the translation of such frameworks requires that barriers to implementation be addressed in the formative research stages. Transparency, accountability, suitability, and adaptability may be bolstered by clinician engagement through a direct empathetic approach aimed at determining complex user requirements of implementation, usability, and workflow integration. ObjectivesWe aimed to utilize a structured user-centered design approach (double-diamond design framework) to 1) qualitatively explore clinicians experience with the current asthma management system, 2) identify user requirements to improve algorithm explainability and A-GPS prototype, and 3) identify potential barriers to ML-based CDS system use. MethodsAt the discovery phase, we first shadowed to understand the practice context. Then, semi-structured interviews were conducted online with 14 clinicians who provide asthma care at two outpatient facilities. Participants were asked about their current difficulties in gathering information for pediatric asthma patients, their expectations of ideal workflows and tools, and suggestions on user-centered interfaces and features. At the define phase, a synthesis analysis was conducted to converge key results from interviewees insights into themes, eventually forming critical how might we research questions to guide model development and implementation. ResultsWe identified user requirements and potential barriers associated with three overarching themes: 1) Usability and Workflow Aspects of the ML System, 2) User Expectations and Algorithm Explainability, and 3) Barriers to Implementation in Context. Even though the responsibilities and workflows vary among different roles, the core asthma-related information and functions they requested were highly cohesive, which allows for a shared information view of the tool. Clinicians hope to perceive the usability of the model with the ability to note patients high risks and take proactive actions to manage asthma efficiently and effectively. For optimal machine-learning algorithm explainability, requirements included documentation to support the validity of algorithm development and output logic, and a request for increased transparency to build trust and validate how the algorithm arrived at the decision. Acceptability, adoption, and sustainability of the asthma management tool are implementation outcomes that are reliant on the proper design and training as suggested by participants. ConclusionsAs part of our comprehensive informatics-based process centered on clinical usability, we approach the problem using a theoretical framework grounded in user experience research leveraging semi-structured interviews. Our focus on meeting the needs of the practice with machine learning technology is emphasized by a user-centered approach to clinician engagement through upstream technology design.