Significance tests for Leave-One-Out Inter-Subject Correlations in fMRI

The inter-subject correlation (ISC) of fMRI data of different subjects performing the same task is a powerful way to localize and differentiate neural processes caused by a stimulus from those that spontaneously or idiosyncratically take place in each subject. The wider adoption of this method has however been impeded by the lack of widely available tools to assess the significance of the observed correlations. Several non-parametric approaches have been proposed, but these approaches are computationally intensive, challenging to implement, and sensitive methods to correct for multiple comparison across voxels are not yet well established. More widely available, and computationally simple, parametric methods have been criticized theoretically on the basis that dependencies in the data could inflate false positives. Here, we therefore endeavored to assess the actual performance of parametric tests on leave-one-out ISC values in two ways. First, we assess whether parametric tests protect against Type I error by assessing how often they find significant clusters of synchronized activity in publicly available datasets, in which such synchronization should not occur. This includes three resting state datasets, and one dataset in which participants did view movies, but where we randomly select segments that were not taken at the same time in the same movie. Contrary to what has been suspected, we find that parametric tests with corrections for multiple comparisons do protect appropriately against Type I error in that data. This was true for FDR correction at the voxel level at q < 0.05, with a minimum cluster-size of k = 20 voxels, FWE correction at the voxel level at < 0.05, with a minimum cluster-size of k = 5, and for correction at the cluster-level with punc < 0.001 with k = max(20, FWEc). Second, we assessed how these parametric tests compare with non-parametric methods when it comes to detecting ISC when participants actually did watch the same movies. We used a dataset including 150 participants viewing two movies, and used a bootstrapping thresholding of the ISC in the entire dataset to outline our best guess of the network of brain regions that truly synchronize while viewing the movies. We then drew subsamples of between 10 and 50 participants from the entire dataset, calculated the ISC, and thresholded it using our candidate methods. We find that FDR thresholding with k = 20 in particular, was substantially more sensitive than bootstrapping methods in detecting this network even in smallish samples of N = 20 participants typical of cognitive neuroscience studies, while at the same time retaining appropriate specificity. Because the parametric tests we show to perform well are more readily available to the neuroscience community than the non-parametric tests previously championed, we trust that this finding paves the way to a wider adoption of ISC, and empowers a wider range of neuroimagers to use ISC to tackle the challenges of naturalistic neuroscience. In particular in the context of often limited sample sizes and modest effect sizes in cognitive neuroscience, we trust that using FDR correction in particular will help neuroimagers identify the contribution of higher brain regions that process stimuli in more loosely timed fashions, more effectively than non-parametric alternatives.