SHAPE AI: Development and Expert Validation of a Survey for Human AI Performance Evaluation in Healthcare

ObjectiveTo develop and content-validate a brief, expert-informed Survey for Human-AI Performance Evaluation (SHAPE-AI) for near-real-time assessment of how clinical AI affects human performance. BackgroundAI-enabled clinical decision support can improve outcomes only when aligned with clinician workflows, and cognitive demands. Existing evaluations measure technical performance and adoption, providing limited assessment of how AI shapes human performance. There is a lack of concise, operationally feasible instruments to measure AI impact on these human factors outcomes in clinical settings. MethodWe used a construct-driven, multi-stage development process. A literature review and prior qualitative work with users of a deployed sepsis prediction tool identified core human performance constructs. Preliminary items were created and iteratively refined through two expert panels. Six clinical informatics experts evaluated representativeness and clarity using content validity indices (CVI). Seven human factors experts then refined constructs, item wording, and response formats through ratings and focus groups, emphasizing discriminant validity, cognitive bias mitigation, and feasibility for deployment within 24 hours of AI use. ResultsA concise 10-item instrument was created, comprising perceived impact, interpretability, agreement with the AIs findings, agreement with the AIs recommendations, trust, workload, provider-patient and provider-team relationships, unexpected outcomes. ConclusionThe SHAPE-AI instrument is a theoretically grounded, operationally feasible tool to monitor human performance as relates to AI use. ApplicationHealth care organizations can deploy SHAPE-AI as a rapid, standardized probe to detect workflow misalignment, mis-calibrated reliance, communication disruptions, and unintended consequences of AI, informing safer design, implementation, and optimization of clinical AI tools. PrecisSHAPE-AI is a brief, expert-validated survey designed to capture clinicians near-real-time perceptions of how AI-enabled decision support affects human performance such as their situational awareness, decision-making, workload, and trust. SHAPE-AI offers health systems a practical, standardized way to monitor and understand the impact of AI on human performance.