Smile-to-Bert: a pre-trained Transformer model for molecular property prediction using SMILES representations

Molecular property prediction is crucial for drug discovery. Over the years, deep learning models have been widely used for these tasks; however, large datasets are often needed to achieve strong performances. Pre-training models on vast unlabeled data has emerged as a method to extract contextualized embeddings that boost performance on smaller datasets. The Simplified Molecular Input Line Entry System (SMILES) encodes molecular structures as strings, making them suitable for natural language processing. Transformers, known for capturing long-range dependencies, are well suited for processing SMILES. One such transformer-based architecture is Bidirectional Encoder Representations from Transformers (BERT), which only uses the encoder part of the Transformer and performs classification and regression tasks. Pre-trained transformer-based architectures using SMILES have significantly improved predictions on smaller datasets. Public data repositories such as PubChem, which provide SMILES, among other data, are essential for pre-training these models. SMILES embeddings that combine chemical structure and physicochemical property information could further improve performance on tasks such as Absorption, Distribution, Metabolism, Excretion, and Toxicity prediction. To this end, we introduce Smile-to-Bert, a pre-trained BERT architecture designed to predict 113 RDKit-computed molecular descriptors from PubChem SMILES. This model generates embeddings that integrate both molecular structure and physicochemical properties. We evaluate Smile-to-Bert on 22 datasets from the Therapeutics Data Commons and compare its performance with that of the 2-encoder model and a Transformer model. Smile-to-Bert achieves the best result on one dataset, while the combination of Smile-to-Bert with the other models leads to improved performance on 8 datasets. Additionally, the state-of-the-art Transformer is applied to Absorption, Distribution, Metabolism, Excretion, and Toxicity prediction for the first time, achieving the best performance on the Therapeutics Data Commons leaderboard of one dataset. Scientific Contribution: We present Smile-to-Bert, a model pre-trained to predict 113 molecular descriptors directly from SMILES representations. We evaluate our model on 22 property prediction datasets and compare the performance with other language models. We demonstrate that integrating our model with two other pre-trained models improves the performance in 8 datasets.