Clinical validation of large-scale functional assays: insights from 2,120 gene-truthset-assay evaluations

Background: Clear guidance is lacking regarding how 'truthset' variants should be used for clinical validation of functional assays, namely determining the allocatable evidence points (EPs) towards clinical classification. It is argued that assays should be validated using truthsets of missense variants, as this is the variant type for which classification is most impacted by functional data. EPs will be influenced by both the number of available 'truthset' variants and their concordance with assay readouts. Methods: We first reviewed 112 sets of ClinGen gene-specific classification specifications (CSPECs) to assess methodologies they applied for truthset assembly and clinical validation of assays. We then proposed differing rules regarding variant type and stringency of classification by which truthsets might be assembled using ClinVar-extracted classifications. We then examined augmentation of ClinVar-classified truthsets with 'proxy-clinical' benign-classified missense variants systematically assembled applying ACMG/AMP rules (of differing stringencies). In total, these constituted 70 basic approaches to ClinVar-based truthset assembly, which we applied to VHL, BRCA1, BRCA2 and RAD51C. We additionally analysed the impact on the size of the truthsets of changing the specified phenotypes against which ClinVar classification had been submitted. We then applied these truthsets to quantify concordance and allocatable EPs for five large-scale multiplexed functional assays for VHL, BRCA1, BRCA2, and RAD51C. Results The EPs from clinical validation of each assay varied widely according to which truthset was used across 2,120 permutations of gene-truthset-assay combinations. For example, sequentially applying 700 different ClinVar-based truthsets to 2,268 VHL assay variant readouts (70 basic ClinVar-based approaches, augmented by examining 5 different phenotypes for each basic approach, and separate validation against two defined deleterious zones), the evidence strength allocatable for pathogenicity ranged from nil to strong evidence (0.0 to 5.6 EPs); for benignity it ranged from supporting to strong evidence (-1.5 to -6.4 EPs). Clinical validation using truthsets comprising just ClinVar-classified missense variants typically resulted in lower EPs than truthsets comprising protein truncating (PTV) and synonymous variants; this was more due to paucity of ClinVar-classified missense truthset variants than poorer concordance. Augmentation with larger 'proxy-clinical' benign-classified missense truthsets typically improved evidence allocatable for pathogenicity, with improved power negating modest reduction in concordance. Conclusions EPs can be improved by augmentation with systematically-generated 'proxy-clinical' benign-classified missense variants and/or reduction of truthset stringency. Explicit prescriptive clinical guidance is urgently required to improve consistency in clinical validation of functional assays and consequent evidence application for clinical variant classification.