An explanation for the sister repulsion phenomenon in Patterson's f-statistics

Patterson's f-statistics are among the most heavily utilized tools for analyzing genome-wide allele frequency data for demographic inference. Beyond studying admixture, f3- and f4-statistics are also used for clustering populations to identify groups with similar histories. However, previous studies have noted an unexpected behavior of f-statistics: multiple populations from a certain region systematically show higher genetic affinity to a more distant population than to their neighbors, a pattern that is mismatched with alternative measures of genetic similarity. We call this counter-intuitive pattern "sister repulsion". We first present a novel instance of sister repulsion, where genomes from Bronze Age East Anatolian sites show higher affinity toward Bronze Age Greece rather than each other. This is observed both using f3- and f4-statistics, contrasts with archaeological/historical expectation, and also contradicts genetic affinity patterns captured using principal components analysis or multidimensional scaling on genetic distances. We then propose a simple demographic model to explain this pattern, where sister populations receive gene flow from a genetically distant source. We calculate f3- and f4-statistics using simulated genetic data with varying population genetic parameters, confirming that low-level gene flow from an external source into populations from 1 region can create sister repulsion in f-statistics. Unidirectional gene flow between the studied regions (without an external source) can likewise create repulsion. Meanwhile, similar to our empirical observations, multidimensional scaling analyses of genetic distances still cluster sister populations together. Overall, our results highlight the impact of low-level admixture events when inferring demographic history using f-statistics.