Sketching the SRA: experiment-wide similarity search across 4.8 million metagenomes

Recent efforts have demonstrated that systematic analysis of the millions of metagenomes in the Sequence Read Archive (SRA) enables groundbreaking biological discoveries. However, SRA-scale comparison of whole metagenomes has remained elusive. Here, we introduce the most comprehensive resource to date for experiment-to-experiment analysis of all SRA metagenomes, comprising FracMinHash sketches (470GB), hypergen sketches (19GB), and an all-pairs Jaccard similarity matrix (90GB), which we release online at https://github.com/RolandFaure/SRA-metagenome-sketches-databases. These resources enable a wide range of analyses, which we demonstrate with several applications. First, we use the FracMinHash sketches to uncover temporal trends, finding that the amount of newly discovered microbial sequence continues to grow, rather than saturate or shrink. Second, we use the hypergen sketches, which enable nearest-neighbor search, to identify co-assembly candidates and thereby improve assembly quality. Third, we use the similarity matrix to analyze all SRA metagenomes, identifying duplicated submissions and over 50K candidates for metadata correction. Together, our resources unlock systematic analysis of metagenomic data at unprecedented scale.