A Mixed-Methods Evaluation of Clinician Experiences and Adoption Patterns of an EHR-integrated Generative AI-based Clinical Decision Support in Kenya

ObjectiveTo quantify the adoption pattern of an LLM-based clinical decision support system across private primary health facilities in Kenya (operated by Penda Health), and explore factors influencing clinician uptake and overall experience. MethodsA mixed-methods study combining quantitative analysis of CDSS metadata from all consultations that took place between 1st February and 1st October 2024, augmented by qualitative data from 42 staff members (26 clinical officers, 10 facility managers, four nurses, one quality assurance manager, and one business analyst). Data collection included journey mapping interviews (n=7), user-experience interviews (n=25), focus groups (n=2), and system utilization metrics. Quantitative data were summarized using descriptive statistics, and qualitative data analysed using thematic analysis (drawing on established theories of technology adoption and change management). ResultsIn total, there were 258,106 clinical episodes across all Penda Health facilities over the 8-month observation period, of which 56,050 (21.7%) were augmented by use of the AI Consult. AI Consult use, aggregated across the 16 facilities, increased from 4% to 47% over 8 months. Feedback on the CDSS guidance was infrequent (only being provided in 31% of cases), but when it was, it was overwhelmingly positive (99.5%). The qualitative investigation identified five key themes associated with clinicians experiences with the AI Consult tool: (1) there are several value propositions to an AI Consult style tool, (2) Clinicians confidence in the AI consult grew with time, (3) clinicians application of the AI consult is influenced by case complexity, (4) responses from the AI consult are largely believed and valued by clinicians but several improvements are recommended, and (5) clinicians find the AI consult easy to use but identified several pain points that warrant attention. DiscussionSuccessful GenAI/LLM-enhanced CDSS implementation in resource-constrained settings requires: (1) robust technological infrastructure, (2) localisation to reflect clinical guidelines, (3) structured change management with clinical champions, and (4) seamless workflow integration. Future product development exercises should specifically consider alternatives to active solicitation of CDSS input, as it is liable to overconfidence-related underutilization. Key Questions Box (BMJ Digital Health and AI)O_ST_ABSWhat is already known?C_ST_ABSO_LIAI-powered clinical decision support systems face implementation challenges, especially in low-resource contexts. C_LIO_LIStudies from high-income countries suggest that workflow integration and perceived usefulness are key drivers of adoption. C_LIO_LILimited research exists on the implementation of GenAI in resource-constrained healthcare environments. C_LI What are the new findings?O_LIIt is possible to achieve non-trivial adoption of GenAI-based CDSS (4% to 47% over 8 months) when supported by structured change management. C_LIO_LIThere are several value propositions for a GenAI-based CDSS, including improved quality of care and serving as a learning aid. C_LIO_LISeveral factors influenced utilization/engagement, from perceived case complexity to ease of use and trust in the system. C_LIO_LIClinicians behaviour and interactions with GenAI tools (as demonstrated by varying prompt use) mature over time based on perceived usefulness. C_LI What do the new findings imply?O_LISustainable AI implementation in low-resource settings requires addressing both technical infrastructure and human factors. C_LIO_LIRelying on active solicitation of inputs from an AI system is likely to suffer from overconfidence bias, which limits its overall utility, and instead, we should explore other methods of integrating CDSS into the relevant clinical workflows. C_LI