Evolution under Stochastic Transmission: Mutation-Rate Modifiers

In evolutionary models of large populations, it is common to analyze the effects of cyclic or random variation in the parameters that describe selection. It is less common, however, to study how stochasticity in the genetic transmission process itself affects evolutionary outcomes. Suppose that a gene locus has alleles A and a under constant selection. This locus is linked to a modifier locus with alleles M1 and M2, which control the mutation rate from A to a. The Reduction Principle states that, near a mutation-selection balance where M1 is fixed with mutation rate u1, a rare allele M2 can invade if its associated rate u2 is lower than u1. This result, valid for both haploids and diploids, assumes constant mutation rates through time. We extend this framework by allowing the mutation rate associated with M2 to fluctuate randomly across generations, denoted as u2,t. In this stochastic setting, the condition for invasion by a new modifier allele depends not only on the resident mutation rate u1 and the mean mutation rate u2 associated with the invading allele, but also on the temporal distribution of u2,t, the strength of selection at the A/a locus, and the recombination rate between M1/M2 and A/a. The analysis shows how stochasticity and recombination in transmission do not simply modify the magnitude of evolutionary change predicted under deterministic assumptions. Instead, through their interaction with selection and linkage, they can generate conditions under which the direction of modifier evolution is qualitatively reversed relative to the deterministic Reduction Principle.