Transformers Enhance the Predictive Power of Network Medicine

BackgroundSelf-attention mechanisms and token embeddings behind transformers allow the extraction of complex patterns from large datasets, and enhance the predictive power over traditional machine learning models. Yet, being trained to make predictions about individual cells or genes, it is not clear if transformers can learn the inherent interaction patterns between genes, ultimately responsible for their mechanism of action. We use Geneformer, pretrained on single-cell transcriptomes, to ask if transformers can implicitly capture molecular dependencies, including protein-protein interactions (PPIs), allowing us to explore the use of transformers to improve network medicine tasks such as disease gene identification and drug repurposing. MethodsWe extracted the cosine similarity of gene embeddings and the attention weights contained in Geneformer, allowing us to test if these weights capture experimentally validated protein interactions. Using dilated cardiomyopathy as a case study, we evaluated the effectiveness of the resulting weighted networks in disease module detection and drug repurposing. ResultsWe found that Geneformer displays awareness of experimentally documented Protein-Protein Interactions, exhibiting higher cosine similarity and attention weights for gene pairs with physical interactions. Weighting PPI networks with the cosine similarity and attention weights improved the detection of disease-associated genes and the accuracy of drug repurposing predictions for dilated cardiomyopathy, surpassing the accuracy of unweighted networks. Finally, we find that combining attention weights and cosine similarities with ranking methods enhances drug candidate prioritization for drug repurposing. ConclusionsWe find that transformers, by implicitly learning the interactions between genes, offer a promising pathway for advancing medicine and drug discovery when integrated with the graph theoretic algorithms used in network medicine.