FST and kinship for arbitrary population structures I: Generalized definitions

FST is a fundamental measure of genetic differentiation and population structure, currently defined for subdivided populations. FST in practice typically assumes independent, non-overlapping subpopulations, which all split simultaneously from their last common ancestral population so that genetic drift in each subpopulation is probabilistically independent of the other subpopulations. We introduce a generalized FST definition for arbitrary population structures, where individuals may be related in arbitrary ways, allowing for arbitrary probabilistic dependence among individuals. Our definitions are built on identity-by-descent (IBD) probabilities that relate individuals through inbreeding and kinship coefficients. We generalize FST as the mean inbreeding coefficient of the individuals local populations relative to their last common ancestral population. We show that the generalized definition agrees with Wrights original and the independent subpopulation definitions as special cases. We define a novel coancestry model based on \"individual-specific allele frequencies\" and prove that its parameters correspond to probabilistic kinship coefficients. Lastly, we extend the Pritchard-Stephens-Donnelly admixture model in the context of our coancestry model and calculate its FST. To motivate this work, we include a summary of analyses we have carried out in follow-up papers, where our new approach has been applied to simulations and global human data, showcasing the complexity of human population structure, demonstrating our success in estimating kinship and FST, and the shortcomings of existing approaches. The probabilistic framework we introduce here provides a theoretical foundation that extends FST in terms of inbreeding and kinship coefficients to arbitrary population structures, paving the way for new estimators and novel analyses.\n\nNote: This article is Part I of two-part manuscripts. We refer to these in the text as Part I and Part II, respectively.\n\nPart I: Alejandro Ochoa and John D. Storey. \"FST and kinship for arbitrary population structures I: Generalized definitions\". bioRxiv (10.1101/083915) (2019). https://doi.org/10.1101/083915. First published 2016-10-27.\n\nPart II: Alejandro Ochoa and John D. Storey. \"FST and kinship for arbitrary population structures II: Method of moments estimators\". bioRxiv (10.1101/083923) (2019). https://doi.org/10.1101/083923. First published 2016-10-27.