Trait evolution with incomplete lineage sorting and gene flow: the Gaussian Coalescent model

Most phylogenetic comparative methods use a species-level phylogeny, ignoring the effect of incomplete lineage sorting (ILS) and hemiplasy on the traits of interest. We consider here a trait controlled additively by one or more unknown loci. Their gene trees may differ from the species phylogeny due to ILS, as modeled by the coalescent process. If the species phylogeny is a network, this process also accounts for gene flow, admixture or hybridization. Our model allows for polymorphism in the ancestral population at the root of the species phylogeny, and predicts heritable within-population variation due to ILS. Even if each locus evolves according to a Brownian motion, the joint distribution of all trait measurements is not generally Gaussian due to ILS. We provide a Gaussian approximation, named the Gaussian Coalescent, and show how to compute its variance matrix efficiently using a single traversal of the species phylogeny. In simulations, this model is much more accurate than the model ignoring ILS. In simulations and on a data set of tomato floral traits, it is favored over the standard Brownian motion model with extra within-population variance. The GC model opens new avenues for various phylogenetic comparative methods, accounting for hemiplasy and gene flow simultaneously. It is implemented in phylolm v2.7.0 and in PhyloTraits v1.2.0.