A cross-attention transformer encoder for paired sequence data
Dens, C.; Laukens, K.; Meysman, P.; Bittremieux, W.
Show abstract
Transformer-based sequence encoding architectures are often limited to a single-sequence input while some tasks require a multi-sequence input. For example, the peptide-MHCII binding prediction task where the input consists of two protein sequences. Current workarounds to solve this input-type mismatch lack resemblance with the biological mechanisms behind the task. As a solution, we propose a novel cross-attention transformer encoder that creates a cross-attended embedding of both input sequences. We compare its classification performance on the peptide-MHCII binding prediction task to a baseline logistic regression model and a default transformer encoder. Finally, we make visualizations of the attention layers to show how the different models learn different patterns.
Matching journals
The top 7 journals account for 50% of the predicted probability mass.
Similar papers in this journal
Similar papers in this journal
- Multi-Head Attention-based U-Nets for Predicting Protein Domain Boundaries Using 1D Sequence Features and 2D Distance Maps 96%
- A Comparison of Embedding Aggregation Strategies in Drug-Target Interaction Prediction 95%
- Struct2Graph: A graph attention network for structure based predictions of protein-protein interactions 95%
Similar papers in this journal
- MULAN: Multimodal Protein Language Model for Sequence and Structure Encoding 96%
- GRU-SCANET: Unleashing the Power of GRU-based Sinusoidal CApture Network for Precision-driven Named Entity Recognition 95%
- KSMoFinder - Knowledge graph embedding of proteins and motifs for predicting kinases of human phosphosites 95%
Similar papers in this journal
- A novel interpretable deep transfer learning combining diverse learnable parameters for improved T2D prediction based on single-cell gene regulatory networks 95%
- DeepInsight-3D for precision oncology: an improved anti-cancer drug response prediction from high-dimensional multi-omics data with convolutional neural networks 94%
- Unsupervised generative and graph representation learning for modelling cell differentiation 94%
Similar papers in this journal
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.