Zero-shot retention time prediction for unseen post-translational modifications with molecular structure encodings

Mass spectrometry-based proteomics relies on accurate peptide property prediction models to enhance peptide identification and characterization, especially when dealing with peptidoforms. However, current approaches are limited in their ability to generalize to peptides with novel post-translational modifications (PTMs) due to insufficient training data. To address this challenge, we introduce MoSTERT (Molecular Structure Transformer Encoder for Retention Time prediction) and its enhanced variant, MoSTERT-2S, two transformer-based models designed for zero-shot prediction of retention times of peptides with unseen PTMs. Unlike conventional models, MoSTERT encodes peptide residues at the molecular structure level, allowing it to handle diverse PTMs. MoSTERT-2S further improves accuracy by employing a two-step strategy: first predicting the retention time of the unmodified peptide, then estimating the retention time shift induced by the PTMs. Evaluation on an external dataset demonstrates that MoSTERT-2S achieves state-of-the-art performance, reducing prediction errors compared to existing methods. Its ability to accurately predict retention times for peptides with a wide variety of PTMs not seen during training highlights its potential for advancing proteomic workflows analyzing proteoforms and protein modifications.