Accurate Skin Lesion Classification Using Multimodal Learning on the HAM10000 Dataset

BackgroundOur aim is to demonstrate that multimodal deep learning can enhance the accuracy of classifying skin lesions using both images and textual descriptions (e.g., demographics, clinical information) compared to a classifier that learn only on images. MethodsWe used the HAM10000 and ISIC 2017 datasets in our study containing 10,000 and 2,750 skin lesion images, respectively. We combined the images with patients data (e.g., sex, age, lesion location) for training and evaluating a multimodal deep learning classification model. The dataset was split into 70% for training the model, 20% for the validation set, and 10% for the testing set. We compared the multimodal models performance to well-known deep learning models that only use images for classification. ResultsWe used accuracy and area under the curve (AUC) receiver operating characteristic (ROC) as the metrics to compare the models performance. Our multimodal model outperformed the competitors and achieved the best results. Our models accuracy and AUCROC was 0.9411 and 0.9426, respectively, on HAM10000. On ISIC 2017, our models accuracy and AUCROC was 0.7971 and 0.8253, respectively. ConclusionOur study showed that a multimodal deep learning model can outperform traditional deep learning models for skin lesion classification on the HAM10000 and ISIC 2017 datasets. Our approach can enable primary care clinicians to screen for skin cancer in patients (residing in areas lacking access to expert dermatologists) with higher accuracy and reliability.