Enlarging the Scope of Randomization and Permutation Tests in Neuroimaging and Neuroscience

Especially for the high-dimensional data collected in neuroscience, nonparametric statistical tests are an excellent alternative for parametric statistical tests. Because of the freedom to use any function of the data as a test statistic, nonparametric tests have the potential for a drastic increase in sensitivity by making a biologically-informed choice for a test statistic. In a companion paper (Geerligs & Maris, 2020), we demonstrate that such a drastic increase is actually possible. This increase in sensitivity is only useful if, at the same time, the false alarm (FA) rate can be controlled. However, for some study types (e.g., within-participant studies), nonparametric tests do not control the FA rate (see Eklund, Nichols, & Knutsson, 2016). In the present paper, we present a family of nonparametric randomization and permutation tests of which we prove exact FA rate control. Crucially, these proofs hold for a much larger family of study types than before, and they include both within-participant studies and studies in which the explanatory variable is not under experimental control. The crucial element of this statistical innovation is the adoption of a novel but highly relevant null hypothesis: statistical independence between the biological and the explanatory variable.