Cross-Attention Enables Context-Aware Multimodal Skin Lesion Diagnosis
Mridha, K.; Islam, H.
Show abstract
Clinical diagnosis of skin lesions integrates visual dermoscopic features with patient context such as age, skin type, and lesion characteristics. However, most artificial intelligence systems for dermoscopic analysis rely solely on image data and ignore structured clinical metadata. We developed a multimodal deep learning framework that combines dermoscopic images with patient metadata and evaluated whether cross-attention mechanisms better capture contextual interactions than conventional fusion strategies. Using 1,568 lesions from the PAD-UFES-20 dataset (69% malignant) with associated metadata (age, sex, Fitzpatrick skin type, anatomical site, and lesion diameter), we compared four models: metadata-only logistic regression, image-only ResNet18, late fusion via feature concatenation, and cross-attention-based fusion. The image-only model achieved strong discrimination (AUC 0.9776), while late fusion slightly reduced performance (AUC 0.9717). The proposed cross-attention model achieved the best overall results (AUC 0.9818, AUPRC 0.9924) with improved calibration (ECE 0.0379). These findings suggest that attention-based multimodal learning enables more effective integration of patient context for automated skin lesion diagnosis.
Matching journals
The top 8 journals account for 50% of the predicted probability mass.
Similar papers in this journal
Similar papers in this journal
- A human-in-the-loop explanation framework for morphologically transparent AI predictions from whole-slide images 93%
- A Deep Learning Based Smartphone Application for Early Detection of Nasopharyngeal Carcinoma Using Endoscopic Images 93%
- Equity-Enhanced Glaucoma Progression Prediction from OCT with Knowledge Distillation 92%
Similar papers in this journal
- Using Adversarial Images to Assess the Stability of Deep Learning Models Trained on Diagnostic Images in Oncology 93%
- Actionability of Synthetic Data in a Heterogeneous and Rare Healthcare Demographic; Adolescents and Young Adults (AYAs) with Cancer 91%
- Histology-based Prediction of Therapy Response to Neoadjuvant Chemotherapy for Esophageal and Esophagogastric Junction Adenocarcinomas Using Deep Learning 89%
Similar papers in this journal
- An Inherently Interpretable AI model improves Screening Speed and Accuracy for Early Diabetic Retinopathy 94%
- Assessing generalizability of an AI-based visual test for cervical cancer screening 94%
- Uncovering the effects of model initialization on deep model generalization: A study with adult and pediatric chest X-ray images 93%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.