ngsAMOVA: A Probabilistic Framework for Analysis of Molecular Variance, dXY and Neighbor-Joining Trees with Low Depth Sequencing Data

MotivationNext-generation sequencing (NGS) has transformed population genetics and evolutionary biology, but the data produced in studies of non-model organisms, ancient DNA, and environmental DNA often consist of low- or medium-depth sequencing. Analyses of these data rely on computational methods that utilize genotype likelihoods (GLs) to account for genotype uncertainty. Nevertheless, many widely-used analysis methods, such as analysis of molecular variance (AMOVA) and methods for estimating phylogenetic trees using nucleotide divergence (dXY) still lack the probabilistic frameworks necessary to accommodate GLs. ResultsWe introduce ngsAMOVA, a novel probabilistic framework for analyzing molecular variation in population hierarchies with low- and medium-depth sequencing data. It employs an Expectation Maximization algorithm to first estimate the joint genotype probabilities for pairs of individuals, accounting for genotype uncertainty using GLs. It then uses these estimates to generate a pairwise distance matrix, which can be used for AMOVA, estimation of dXY, and for estimating phylogenetic trees using Neighbor-Joining. Hypothesis testing is facilitated using genomic block-bootstrapping. Through extensive simulations, we demonstrate that ngsAMOVA provides more accurate results compared to genotype calling at low and medium read depths. Overall, ngsAMOVA represents a methodological advance in the analysis of molecular variance and divergence under sequencing uncertainty. It provides a robust framework, opening up numerous possibilities for gaining insights into the evolutionary histories through its applications. ngsAMOVA is available as a fast, efficient, and user-friendly program written in C/C++. AvailabilityngsAMOVA is freely available at https://github.com/isinaltinkaya/ngsAMOVA. Contactisin.altinkaya@sund.ku.dk Supplementary informationSupplementary data are available online.