Species tree disequilibrium positively misleads models of gene family evolution

Gene duplication is a key source of evolutionary innovation, and multigene families evolve in a birth-death process, continuously duplicating and pseudogenizing through time. To empirically test hypotheses about adaptive expansion and contraction of multigene families across species, models infer gene gain and loss in light of speciation events and these inferred gene family expansions may lead to interpretations of adaptations in particular lineages. While the relative abundance of a gene subfamily in the subgenome may reflect its functional importance, tests based on this expectation can be confounded by the complex relationship between the birth-death process of gene subfamily evolution and the species phylogeny. Using simulations, we confirmed tree heterogeneity as a confounding factor in inferring multi-gene adaptation, causing spurious associations between shifts in birth-death rate and lineages with higher branching rates. We then used the olfactory receptor (OR) repertoire, the largest gene family in the mammalian genome, of different bat species with divergent diets to test whether expansions in olfactory receptors are associated with shifts to frugivorous diets. After accounting for tree heterogeneity, we robustly inferred that certain OR subfamilies exhibited expansions associated with dietary shifts to frugivory. Taken together, these results suggest ecological correlates of individual OR gene subfamilies can be identified, setting the stage for detailed inquiry into within-subfamily functional differences.