Computer Vision Estimation of Stress and Anxiety Using a Gamified Mobile-based Ecological Momentary Assessment and Deep Learning: Research Protocol

Stress and anxiety can contribute to the development of major health problems such as heart disease, depression, and obesity. Due to its subjective nature, it is challenging to precisely measure human affect by relying on automated approaches. We therefore propose a personalized prediction framework fine-tuned for each participant in lieu of the traditional "one-size-fits-all" machine learning approach. We aim to collect such individualized data via two distinct procedures: 1) a smartphone-based ecological momentary assessment of stress, and 2) Zoom calls. The data collected from these periodic self-reports will include selfie photographs and ecological momentary assessments of affect. To enhance user engagement during the data collection process, we propose the use of gamification, an emerging trend which involves influencing user behavior and lifestyle by incorporating fun and engaging game elements into non-game contexts (e.g., health-related tasks). In addition to developing a standardized platform to achieve more accurate recognition of stress and anxiety, we plan to conduct a concurrent study in which we will capture videos of our subjects undertaking the Stroop Color Word and Amygdala Test and analyze the footage to identify additional significant characteristics that relate to anxiety. This could include features such as head and mouth movements, lip and cheek deformations, eye gaze, and blinking rates. The final results will provide a comparative evaluation of both objective measures of stress. This research project was approved by the University of Hawaii Institutional Review Board.