VQ-MAP: a flexible, versatile framework for animal behavior understanding

The brain generates behaviors that reflect configurations and computations of neural systems. Animal pose analysis can comprehensively profile behaviors in terms of stereotyped action expression, but the absence of common tracking standards makes it difficult to relate measurements across labs and experiments, hindering data exchange and collective knowledge building. Here, we present VQ-MAP, a versatile approach capable of mapping 2D or 3D keypoint sequences with divergent pose formats into a common set of discrete behavioral motifs. VQ-MAP utilizes permutation-invariant attention blocks for keypoint-agnostic processing and a quantized latent space for fast, interpretable delineation of motifs organized into type-subtype hierarchies. We deployed VQ-MAP to align and harmonize heterogeneous pose datasets, co-profiling rodent behaviors in multiple species and models of autism. In longitudinal recordings of rat behavioral maturation, VQ-MAP revealed developmental trajectories connecting specific behaviors across periods of substantial bodily growth and change. By unifying heterogeneous behavioral measurements, our framework will accelerate neurobiological discovery and help the community exercise the full potential of large-scale behavioral datasets.