Beyond Intensity: Cross-Dataset Consistency of Temporal Facial Action-Unit Dynamics as Transferable Markers of Depression

Facial behavior is a widely studied objective signal for depressive-symptom analysis, yet most systems are trained and evaluated within a single dataset, leaving it unclear whether the learned representations reflect depression or dataset-specific artifacts. Prior work has also relied largely on aggregate intensity statistics and on small samples. We reframe the problem by asking which facial action unit (AU) features transfer across datasets, rather than which maximize within-dataset discrimination. Using time-resolved AU dynamics in a Korean cohort of 2,608 participants, including 265 with PHQ-9-defined depressive symptoms, , recorded under happy and unhappy emotion-elicitation conditions, asking which AU features transfer across datasets rather than which maximize within-dataset discrimination. From AU time-series extracted with OpenFace, we computed 568 features spanning intensity, temporal, dynamic, peak-structure, and Duchenne (genuine smile) co-activation patterns. We then tested their directional transferability on the US DAIC-WOZ dataset, which differs in race, language, interview task, recording length, and recording setting. Within the source cohort, peak-interval features gave the strongest discriminative signal (AU26 peak-interval, Cohen's d = -0.66), yet reversed sign externally, whereas slower temporal features and AU06 (cheek raiser) preserved their direction despite modest effect sizes. Excluding peak features raised directional agreement from 56-64% to 82-90%. Within-dataset discriminative strength therefore does not predict cross-dataset transferability, supporting directional consistency as a practical criterion for selecting transferable affective-model inputs. Smile analysis further showed that depressive symptoms were marked less by reduced smiling than by eye-mouth decoupling (d = -0.41), which survived covariate adjustment and matched the voluntary-emotional facial-pathway distinction. Being low-dimensional and non-identifying, AU time series may support privacy-conscious, multi-site depression research without sharing raw video.