Heterogeneous reconstruction algorithms for cryoEM achieve limited particle classification accuracy on real benchmark datasets

The emergence of single-particle cryoEM as a powerful method for structure determination has in large part been fueled by its ability to resolve both single static structures and complex conformational landscapes. Indeed, modern approaches to the heterogeneous reconstruction task can resolve 100s-1,000s of different maps from a single cryoEM dataset. How accurate these algorithms are, however, has proven difficult to rigorously assess, due to a lack of suitable benchmark datasets containing both realistic noise features and ground-truth labels. To address this obstacle, we recently developed a series of benchmark datasets that leverage the targeting power of Cas9 and the programmable heterogeneity of DNA to newly offer access to ground-truth per-particle structural labels in real data. Here, we challenged two popular heterogeneous reconstruction algorithms with mixed particle stacks resampled in silico from these datasets, finding that existing approaches resolve the encoded heterogeneity with limited accuracy. In particular, in realistic particle stacks with complex, multi-scale, and multi-axis heterogeneity, we observed that reconstruction of encoded heterogeneity depended strongly on the application of prior information about where heterogeneity was expected, and that individual particle assignments were made with significant error even when the correct structural states were reconstructed. Both molecular breathing motions and data collection features, such as defocus and projection angle, contributed to the observed particle assignment error. These results highlight important shortcomings of existing heterogeneous reconstruction methods and suggest new avenues for method development in both data collection strategies and in heterogeneous classification and reconstruction algorithms.