Threshold-Free Network-Oriented Statistics in Neuroscience

Network neuroscience has emerged as an indispensable tool for studying brain structure and function. Currently, the network-based statistic (NBS) procedure is widely used for dealing with massive multiple testing/comparison problems in brain networks. However, the NBS requires choosing a hard cluster-forming threshold, lacking objective rules. A powerful and flexible statistical framework is urgently needed with growing interest in finer-grained network explorations across modalities and scales. Here, we introduce a permutation-based framework--"Threshold-Free Network-Oriented Statistics" (TFNOS). It integrates two "threshold-free" pathways: traversing all cluster-forming thresholds (TT) and using predefined clusters (PC). The TT procedure, building upon the threshold-free network-based statistics, requires setting additional parameters. The PC procedures comprise six variants given the degree of freedom in pooling data, null distribution construction, and controlled error rate. Using numerical simulations, we evaluated the performance of the TT procedure under 600 parameter combinations, then benchmarked TFNOS procedures and baselines across different topologies of effects, sample sizes, and effect sizes, and finally provided illustrative examples with real data. We offer recommended parameter values that allow the TT procedure to stably maintain leading power, while empirically controlling the false discovery rate (FDR) beyond only weakly controlling the familywise error rate (FWER). Notably, the relevant parameters commonly employed in the field appear overly liberal. Furthermore, for the PC procedures, FDR-controlling variants showed improved power compared to FWER-controlling variants, and some of them are simple but do not compromise power. The nonparametric PC procedures allow the selection of any test statistics considered appropriate. Overall, the TFNOS is a generalized framework for inference on edges/nodes of undirected/directed brain networks. We provide empirical and principled criteria for selecting appropriate procedures and may enhance the reproducibility and sensitivity of future brain research.