Gene genealogies in haploid populations evolving according to sweepstakes reproduction

Recruitment dynamics, or the distribution of the number of offspring among individuals, is fundamental to ecology and evolution. We take sweepstakes reproduction to mean a skewed (heavy right-tailed) offspring number distribution without natural selection being involved. Sweepstakes may be generated by chance matching of reproduction with favorable environmental conditions. Gene genealogies generated by sweepstakes reproduction are in the domain of attraction of multiple-merger coalescents where a random number of lineages merges at such times. We consider population genetic models of sweepstakes reproduction for haploid panmictic populations of both constant (N), and varying population size, and evolving in a random environment. We construct our models so that we can recover the observed number of new mutations in a given sample without requiring strong assumptions regarding the population size or the mutation rate. Our main results are (i) continuous-time coalescents that are either the Kingman coalescent or specific families of Beta- or Poisson-Dirichlet coalescents; when combining the results the parameter of the Beta-coalescent ranges from 0 to 2, and the Beta-coalescents may be incomplete due to an upper bound on the number of potential offspring an arbitrary individual may produce; (ii) in large populations we measure time in units proportional to either N/log N or N generations; (iii) incorporating fluctuations in population size leads to time-changed multiple-merger coalescents where the time-change does not depend on ; (iv) using simulations we show that in some cases approximations of functionals of a given coalescent do not match the ones of the ancestral process in the domain of attraction of the given coalescent; (v) approximations of functionals obtained by conditioning on the population ancestry (the ancestral relations of all gene copies at all times) are broadly similar (for the models considered here) to the approximations obtained without conditioning on the population ancestry.