Compression strategies for large-scale electrophysiology data

AbstractWith the rapid adoption of high-density electrode arrays for recording neural activity, electrophysiology data volumes within labs and across the field are growing at unprecedented rates. For example, a one-hour recording with a 384-channel Neuropixels probe generates over 80 GB of raw data. These large data volumes carry a high cost, especially if researchers plan to store and analyze their data in the cloud. Thus, there is a pressing need for strategies that can reduce the data footprint of each experiment. Here, we establish a set of benchmarks for comparing the performance of various compression algorithms on experimental and simulated recordings from Neuropixels 1.0 (NP1) and 2.0 (NP2) probes. For lossless compression, audio codecs (FLAC and WavPack) achieve compression ratios 6% higher for NP1 and 10% higher for NP2 than the best general-purpose codecs, at the expense of a slower decompression speed. For lossy compression, the WavPack algorithm in "hybrid mode" increases the compression ratio from 3.59 to 7.08 for NP1 and from 2.27 to 7.04 for NP2 (compressed file size of[~] 14% for both types of probes), without adverse effects on spike sorting accuracy or spike waveforms. Along with the tools we have developed to make compression easier to deploy, these results should encourage all electrophysiologists to apply compression as part of their standard analysis workflows.