The collapse of genetic incompatibilities in a hybridizing population

Diverging species are often genetically incompatible upon hybridization. Such incompatibilities are considered important in keeping the integrity of species from the disruption of hybrids. However, recent empirical work has shown that not all incompatibilities are gene-flow-proof, and they can collapse due to continuing hybridization. Counterintuitively, many studies found that incompatible alleles are already segregating within species, whereas they should go extinct quickly in a randomly mating population. Due to the complexity of multilocus epistasis, few general principles explain behaviors of incompatibilities under gene flow both within and between species. In the current work, we argue that the redundancy of genetic mechanisms can robustly determine the dynamics of intrinsic incompatibilities under gene flow. While higher genetic redundancy decreases the stability of incompatibilities during hybridization, it also increases the tolerance of incompatibility polymorphism within each species. We treat two general classes of incompatibilities. In the redundant class, similar to the classical Dobzhansky-Muller system, the collapse is continuous and eventually approaches quasi-neutral polymorphism between broadly-sympatric species, often as a result of isolation-by-distance. In the non-redundant class, analogous to the shifting-balance process, incompatibilities collapse abruptly with spatial traveling waves. We obtained simulated and analytical results for several incompatibility models to demonstrate the differences between the two classes. As both redundant and non-redundant genetic mechanisms of incompatibilities are common, the proposed conceptual framework may help understand the abundance of incompatibilities in natural populations.