FERRET: Framework to Evaluate Robustness in Regulatory Networks Using Heterogeneous Cell Types

Techniques for evaluating gene regulatory network (GRN) inference methods typically focus on recovering small ground-truth networks or on benchmarking against simulated data. However, both approaches have important limitations and fail to capture the biological variability present in real datasets. FERRET is a framework for benchmarking single-cell GRN inference methods based on a simple biological assumption: independent estimates of the regulatory network from the same cellular state should resemble one another more closely than estimates from distinct cellular states. Rather than relying on incomplete or simulated ground truth, FERRET quantifies network robustness using two complementary metrics: Robustness Area Under the Curve (RAUC), an AUC-like measure of within-cell-type network similarity relative to between-cell-type similarity, and Monotonicity, which assesses the consistency of network similarity across edge-weight cutoffs. FERRET also supports biological validation through pathway enrichment analysis. We validate FERRET using experimentally derived ChIP-seq networks from B lymphocytes and fibroblasts as positive controls and randomly generated networks as negative controls, showing that biologically related networks receive high robustness scores whereas randomly generated networks receive scores consistent with chance. Finally, we apply FERRET to multiple GRN inference methods on real single-cell RNA-sequencing datasets to identify methods that produce the most robust, biologically informative regulatory networks. GRAPHICAL ABSTRACT O_FIG O_LINKSMALLFIG WIDTH=200 HEIGHT=112 SRC="FIGDIR/small/739795v1_ufig1.gif" ALT="Figure 1"> View larger version (43K): org.highwire.dtl.DTLVardef@da7e4eorg.highwire.dtl.DTLVardef@9a5425org.highwire.dtl.DTLVardef@a6e60org.highwire.dtl.DTLVardef@d466cd_HPS_FORMAT_FIGEXP M_FIG C_FIG