Using Runs of Homozygosity and Machine Learning to Disentangle Sources of Inbreeding and Infer Self-Fertilization Rates

Runs of homozygosity (ROHs) are indicative of elevated homozygosity and inbreeding due to mating of closely related individuals. Self-fertilization can be a major source of inbreeding which elevates genome-wide homozygosity and thus should also create long ROHs. While ROHs are frequently used to understand inbreeding in the context of conservation and selective breeding, as well as for consanguinity of populations and their demographic history, it remains unclear how ROH characteristics are altered by selfing and if this confounds expected signatures of inbreeding due to demographic change. Using simulations, we study the impact of the mode of reproduction and demographic history on ROHs. We apply random forests to identify unique characteristics of ROHs, indicative of different sources of inbreeding. We pinpoint distinct features of ROHs that can be used to better characterize the type of inbreeding the population was subjected to and to predict outcrossing rates and complex demographic histories. Using additional simulations and four empirical datasets, two from highly selfing species and two from mixed-maters, we predict the selfing rate and validate our estimations. We find that self-fertilization rates are successfully identified even with complex demography. Population genetic summary statistics improve algorithm accuracy particularly in the presence of additional inbreeding, e.g. from population bottlenecks. Our findings highlight the importance of ROHs in disentangling confounding factors related to various sources of inbreeding and demonstrate situations where such sources cannot be differentiated. Additionally, our random forest models provide a novel tool to the community for inferring selfing rates using genomic data.