Dynamic modulation of social gaze by sex and familiarity in marmoset dyads

Social communication relies on the ability to perceive and interpret the direction of others attention, and is commonly conveyed through head orientation and gaze direction in humans and nonhuman primates. However, traditional social gaze experiments in nonhuman primates require restraining head movements, significantly limiting their natural behavioral repertoire. Here, we developed a novel framework for accurately tracking facial features and three-dimensional head gaze directions of multiple freely moving common marmosets (Callithrix jacchus). By combining deep learning-based computer vision tools with triangulation algorithms, we were able to track the facial features of marmoset dyads within an arena. This method effectively generates dynamic 3D geometrical facial frames while overcoming common challenges like occlusion. To detect the head gaze direction, we constructed a virtual cone, oriented perpendicular to the facial frame. Using this pipeline, we quantified different types of interactive social gaze events, including partner-directed gaze and joint gaze to a shared spatial location. We observed clear effects of sex and familiarity on both interpersonal distance and gaze dynamics in marmoset dyads. Unfamiliar pairs exhibited more stereotyped patterns of arena occupancy, more sustained levels of social gaze across social distance, and increased social gaze monitoring. On the other hand, familiar pairs exhibited higher levels of joint gazes. Moreover, males displayed significantly elevated levels of gazes toward females faces and the surrounding regions, irrespective of familiarity. Our study reveals the importance of two key social factors in driving the gaze behaviors of a prosocial primate species and lays the groundwork for a rigorous quantification of primate behaviors in naturalistic settings.