Divergence and evolution of assortative mating in a polygenic trait model of speciation with gene flow

Assortative mating is an important driver of speciation in populations with gene flow and is predicted to evolve under certain conditions in few-locus models. However, the evolution of assortment is less understood for mating based on quantitative traits, which are often characterized by high genetic variability and extensive linkage disequilibrium between trait loci. We explore this scenario for a two-deme model with migration, by considering a single polygenic trait subject to divergent viability selection across demes, as well as assortative mating and sexual selection within demes, and investigate how trait divergence is shaped by various evolutionary forces. Our analysis reveals the existence of sharp thresholds of assortment strength, at which divergence increases dramatically. We also study the evolution of assortment via invasion of modifiers of mate discrimination and show that the ES assortment strength has an intermediate value under a range of migration-selection parameters, even in diverged populations, due to subtle effects which depend sensitively on the extent of phenotypic variation within these populations. The evolutionary dynamics of the polygenic trait is studied using the hypergeometric and infinitesimal models. We further investigate the sensitivity of our results to the assumptions of the hypergeometric model, using individual-based simulations.