Serotonin drives aggression and social behaviours of laboratory mice in a semi-natural environment

Aggression is an adaptive social behaviour crucial for the stability and prosperity of social groups. When uncontrolled, aggression leads to pathological violence that disrupts group structure and individual well-being. The comorbidity of uncontrolled aggression across different psychopathologies makes it a potential endophenotype of mental disorders with the same neurobiological substrates. Serotonin plays a critical role in the regulation of impulsive and aggressive behaviours, and mice lacking brain serotonin, due to the ablation of a rate-limiting enzyme of serotonin synthesis (Tryptophan hydroxylase 2, TPH2), are a potential model of pathological aggression. Home cage monitoring allows for the continuous observation and quantification of social and non-social behaviours in group-housed, freely-moving mice. Using an ethological approach, we investigated the impact of central serotonin ablation on everyday expression of social and non-social behaviours and their correlations in undisturbed, group-living Tph2-deficient and wildtype mice. By training a machine learning algorithm on behavioural time series, "allogrooming", "struggling at feeder" and "eating" emerged as key behaviours dissociating one genotype from the other. Although Tph2-deficient mice showed characteristics of pathological aggression and decreased communication compared to wildtype animals they still showed affiliative behaviours to normal levels. Altogether, such distinct and dynamic phenotype of Tph2-deficient mice influenced the groups structure and the dynamic of its hierarchical organization which emerged later. These aspects were analyzed using social network analysis and the Glicko rating methods. This study demonstrates the importance of the ethological approach for understanding the global impact of pathological aggression on different aspects of life both at the individual and the group level. Home cage monitoring allows the observation of the natural behaviours of the mice in a semi-natural habitat and provides an accurate representation of real-world phenomena and pathological mechanisms. The results of this study provide insights into the neurobiological substrate of pathological aggression and their potential role in complex brain disorders.