The impact of incomplete taxon sampling on inference of gene flow by Bayesian and summary methods using genomic sequence data

Interspecific gene flow is commonly inferred using genomic data under the multispecies coalescent (MSC) model. The rate of gene flow is measured by the expected proportion of immigrants in the recipient population at the time of hybridization/introgression. Incomplete taxon sampling can impact inference of gene flow in multiple ways. First unsampled ghost lineages that are sources of introgression may mislead inference of gene flow in analysis of genomic data from modern species. Second incomplete taxon sampling causes merges of branches on the species phylogeny, which represent populations of different sizes, and complicates the definition and estimation of the introgression probability. We use mathematical analysis and computer simulation to examine the impact of incomplete taxon sampling on inference of gene flow and estimation of its rate using genomic data. We introduce a Bayesian testing approach to select models of gene flow for a species triplet (such as inflow, outflow, and ghost introgression), using the Savage-Dickey density ratio to calculate Bayes factors. We show that the approach has excellent power and specificity. We find that genomic data allow reliable estimation of the proportion of immigrants (rather than the number of immigrants), even when the assumed demographic model is incorrect due to incomplete taxon sampling. When population size differs among species, assuming the same size may lead to seriously biased estimates of the rate of gene flow. The f -branch approach is found to be effective in reducing the number of significant gene-flow events from triplet analyses, providing useful hypotheses for rigorous testing, but often to produce underestimates of the rate of gene flow. Our results highlight the need for improving summary methods to accommodate different population sizes and to infer gene flow between sister lineages.