AI-Generated Responses to Patient's Messages: Effectiveness, Feasibility and Implementation

BackgroundGenerative artificial intelligence (GenAI) in healthcare may reduce administrative burden and enhance quality of care. Large language models (LLMs) can generate draft responses to patient messages using electronic health record (EHR) data. This could mitigate increased workload related to high message volumes. While effectiveness and feasibility of these GenAI tools have been studied in the United States, evidence from non-English contexts is scarce, particularly regarding user experience. ObjectiveThis study evaluated the effectiveness, feasibility and barriers and facilitators of implementing Epics Augmented Response Technology (Art) GenAI tool (Epic Systems Corporation, Verona, WI, USA) in a Dutch academic healthcare setting among a broad range of end users. It explored healthcare professionals (HCP) usage metrics, expectations, and early user experiences. MethodsWe conducted a hybrid type 1 effectiveness-implementation design. HCPs of four clinical departments (dermatology, medical oncology, otorhinolaryngology, and pulmonology) participated in a six-month study. Effectiveness of Art was assessed using efficiency indicators from Epic (including all InBasket users in the hospital) and survey scales measuring well-being and clinical efficiency at three time points: PRE, POST-1 (1 month), and POST-2 (4 months). Feasibility of Art was evaluated through adoption indicators from Epic and survey scales on use and usability. Barriers and facilitators of Art implementation were collected through the survey and thematized using the NASSS framework (Nonadoption, Abandonment, Scale-up, Spread and Sustainability). Results237 unique HCPs generated a total of 8,410 drafts. Review and drafting times were similar for users with and without Art, indicating minimal differences. Perceived clinical efficiency declined significantly from PRE to POST-2, while well-being remained unchanged. Adoption was initially high but decreased over time, averaging 16.7% across departments. Usability and intention-to-use scores also declined significantly. Oualitative findings highlighted time savings, well-structured drafts, and patient-centered language as facilitators. Reported barriers included limited impact on time, low practical utility, content inaccuracies, and style misalignment. ConclusionsThis evaluation of a GenAI tool for patient-provider communication in a non-English academic hospital revealed mixed perceptions of effectiveness and feasibility. High initial expectations contrasted with limited perceived impact on time-savings, well-being and clinical efficiency, alongside declining adoption and usability. Barriers and facilitators revealed contrasting views. These findings underscore the need for a workflow for the handling of user feedback, guidance on clinical responsibilities, along with clear communication about the tools purpose and limitations to manage expectations. Additionally, establishing consensus on a set of quality indicators and their thresholds that indicate when a GenAI tool is sufficiently robust will be critical for responsible scaling of GenAI in clinical practice.