Development of attention-based robust deep learning model for tongue diagnosis by smartphone

Compared with tongue diagnosis using tongue image analyzers, tongue diagnosis by smartphone has great advantages in convenience and cost for universal health monitoring, but its accuracy is affected by the shooting conditions of smartphones. Developing deep learning models with high accuracy and robustness to changes in the shooting environment for tongue diagnosis by smartphone and determining the influence of environmental changes on accuracy are necessary. In our study, a dataset of 9003 images was constructed after image pre-processing and labeling. Next, we developed an attention-based deep learning model (Deep Tongue) for 8 subtasks of tongue diagnosis, including the spotted tongue, teeth-marked tongue, and fissure tongue et al, which the average AUC of was 0.90, higher than the baseline model (ResNet50) by 0.10. Finally, we analyzed the objective reasons, the brightness of the environment and the hue of images, affecting the accuracy of tongue diagnosis by smartphone through a consistency experiment of direct subject inspection and tongue image inspection. Finally, we determined the influence of environmental changes on accuracy to quantify the robustness of the Deep Tongue model through simulation experiments. Overall, the Deep Tongue model achieved a higher and more stable classification accuracy of seven tongue diagnosis tasks in the complex shooting environment of the smartphone, and the classification of tongue coating (yellow/white) was found to be sensitive to the hue of the images and therefore unreliable without stricter shooting conditions and color correction.