Development and Evaluation of Artificial Intelligence-Assisted Decision Support System for Public Health Emergency Classification and Escalation in Kenya

Background Timely assessment, classification, and escalation of public health events are essential for effective outbreak response, yet decision-making after event detection remains challenging because of fragmented guidance and variable interpretation of escalation criteria.To strengthen public health emergency management, Kenya developed the Decision-Making Tool for Public Health Emergencies (DMT-PHE), a framework for event assessment, classification, notification, and escalation. An artificial intelligence (AI)-enabled version, the DMT-PHE AI Agent, was subsequently developed to operationalize the framework through decision support. This study describes the development of the DMT-PHE AI Agent and evaluates its performance, usability, safety, and user acceptability. Methods The DMT-PHE AI Agent was developed using a retrieval-augmented generation architecture supported by a curated knowledge base derived from the validated DMT-PHE framework and related public health guidance. A simulation-based pilot evaluation was conducted among 11 public health professionals who independently assessed three standardized outbreak scenarios. AI-generated recommendations were compared with expert-defined gold standards. Outcomes included concordance, response-action coverage, citation performance, safety, usability, and user acceptability. Results Thirty-three scenario evaluations were completed. The AI Agent achieved an overall weighted concordance score of 0.924, with exact agreement of 90.9% for Public Health Events of Initially Unknown Etiology, 81.8% for Rift Valley fever, and 90.9% for Mpox. Citation support was provided in 78.8% of interactions, with no incorrect citations or major safety concerns identified. The mean System Usability Scale score was 85.2, while participants reported high trust (4.27/5), contextual relevance (4.55/5), and perceived time savings (4.82/5). Conclusions The DMT-PHE AI Agent demonstrated that a nationally validated public health emergency decision framework can be successfully translated into an AI-enabled decision-support system. These findings provide early evidence that AI can augment public health emergency decision-making by delivering structured, transparent, and context-specific recommendations while maintaining human oversight, offering a practical model for operationalizing national public health guidance.