MarmoPose: A Deep Learning-Based System for Real-time Multi-Marmoset 3D Pose Trackingp

The ability to track positions and poses (body parts) of multiple monkeys in a 3D space in real time is highly desired by non-human primate (NHP) researchers in behavioral and systems neuroscience because it allows both analyzing social behaviors among multiple NHPs and performing close-loop experiments (e.g., delivering sensory or optogenetics stimulation during a particular behavior). While a number of animal pose tracking systems have been reported, nearly all published work lacks the real-time analysis capacity. Existing methods for tracking freely moving animals have been developed primarily for rodents which typically move on a 2D space. In contrast, NHPs roam in a 3D space and move at a much faster speed than rodents. We have designed a real-time 3D pose tracking system (MarmoPose) based on deep learning to capture and quantify social behaviors in natural environment of a highly social NHP species, the common marmosets (Callithrix jacchus) which has risen to be an important NHP model in neuroscience research in recent years. This system has minimum hardware requirement and can accurately track the 3D poses (16 body locations) of multiple marmosets freely roaming in their homecage. It employs a marmoset skeleton model to optimize the 3D poses and estimate invisible body locations. Furthermore, it achieves high inference speed and provides an online processing module for real-time closed-loop experimental control based on the 3D poses of marmosets. While this system is optimized for marmosets, it can also be adapted for other large animal species in a typical housing environment with minimal modifications.