Human-Centered Design of an Artificial Intelligence (AI) Monitoring System: The Vanderbilt Algorithmovigilance Monitoring and Operations System (VAMOS)

Background.As the use of AI in healthcare is rapidly expanding, there is also growing recognition of the need for ongoing monitoring of AI after implementation, called algorithmovigilance. Yet, there remain few systems that support systematic monitoring and governance of AI used across a health system. In this study, we describe the human-centered design (HCD) process used to develop the Vanderbilt Algorithmovigilance Monitoring and Operations System (VAMOS). Methods.We assembled a multidisciplinary team to plan AI monitoring and governance at VUMC. We then conducted nine participatory design sessions with diverse stakeholders to develop prototypes of VAMOS. Once we had a working prototype, we conducted eight formative design interviews with key stakeholders to gather feedback on the system. We analyzed the interviews using a rapid qualitative analysis approach and revised the mock-ups. We then conducted a multidisciplinary heuristic evaluation to identify further improvements to the tool. Results.Through an iterative, HCD process, we identified key components needed in AI monitoring systems. We identified specific data views and functionality required by end users across several user interfaces including a performance monitoring dashboard, accordion snapshots, and model-specific pages. We distilled general design guidelines for systems to support AI monitoring throughout its lifecycle. One important consideration is how to support teams of health system leaders, clinical experts, and technical personnel that are distributed across the organization as they monitor and respond to algorithm deterioration. Conclusion.VAMOS enables systematic and proactive monitoring of AI tools in healthcare organizations. Our findings and recommendations can support the design of AI monitoring systems to support health systems, improve quality of care and ensure patient safety.