The Fault in Our Sets: A Mixed Methods Analysis of Clinical Value Set Errors

ObjectiveTo characterize clinical value set issues and identify common patterns of errors. Materials and MethodsWe conducted semi-structured interviews with 26 value set experts and performed root cause analyses of errors identified in electronic health records (EHRs). We also analyzed a random sample of user-reported issues from the Value Set Authority Center (VSAC), developing a categorization scheme for value set errors. Additionally, we audited medication value sets from three sources and assessed the impact of value set variations on a clinical quality measure within Vanderbilts Epic system. ResultsInterviews highlighted ongoing difficulties in value set identification, creation, and maintenance, with significant consequences for clinical decision support (CDS), quality measurement, and patient care. Content analysis indicated that 42% of errors involved missing codes, 14% included extraneous codes, and 40% arose from misinterpretations of value set intent; 72% of these errors were present at creation. The audit revealed errors in 50% of medication value sets, predominantly omissions. The impact analysis demonstrated that value set selection altered a clinical quality measures outcome by 3- to 30-fold. DiscussionValue set errors are widespread and arise from a delineable set of causes. Characterizing patterns of errors allowed us to identify best practices and potential solutions to minimize their frequency. ConclusionBetter tools for finding, authoring, auditing and monitoring value sets are urgently needed.