Visualizing the Costs and Benefits of Correcting P-Values for Multiple Hypothesis Testing in Omics Data

The multiple hypothesis testing problem is inherent in high-throughput quantitative genomic, transcriptomic, proteomic, and other "omic" screens. The correction of p-values for multiple testing is a critical element of quantitative omic data analysis, yet many researchers are unfamiliar with the sensitivity costs and false discovery rate (FDR) benefits of p-value correction. We developed models of quantitative omic experiments, modeled the costs and benefits of p-value correction, and visualized the results with color-coded volcano plots. We developed an R Shiny web application for further exploration of these models which we call the Simulator of P-value Multiple Hypothesis Correction (SIMPLYCORRECT). We modeled experiments in which no analytes were truly differential between the control and test group (all null hypotheses true), all analytes were differential, or a mixture of differential and non-differential analytes were present. We corrected p-values using the Benjamini-Hochberg (BH), Bonferroni, and permutation FDR methods and compared the costs and benefits of each. By manipulating variables in the models, we demonstrated that increasing sample size or decreasing variability can reduce or eliminate the sensitivity cost of p-value correction and that permutation FDR correction can yield more hits than BH-adjusted and even unadjusted p-values in strongly differential data. SIMPLYCORRECT can serve as a tool in education and research to show how p-value adjustment and various parameters affect the results of quantitative omics experiments.