Scaffold-Lab: Critical Evaluation and Ranking of Protein Backbone Generation Methods in A Unified Framework

De novo protein design has undergone a rapid development in recent years, especially for backbone generation, which stands out as more challenging yet valuable, offering the ability to design novel protein folds with fewer constraints. However, a comprehensive delineation of its potential for practical application in protein engineering remains lacking, as does a standardized evaluation framework to accurately assess the diverse methodologies within this field. Here, we proposed Scaffold-Lab benchmark focusing on evaluating unconditional generation across metrics like designability, novelty, diversity, efficiency and structural properties. We also extrapolated our benchmark to include the motif-scaffolding problem, demonstrating the utility of these conditional generation models. Our findings reveal that FrameFlow and RFdiffusion in unconditional generation along with Rfdiffusion and GPDL in conditional generation showcased the most outstanding performances. Furthermore, we described a systematic study to investigate conditional generation and applied it to the motif-scaffolding task, offering a novel perspective for the analysis and development of conditional protein design methods. All data and scripts will be available at https://github.com/Immortals-33/Scaffold-Lab.