Assessing the accuracy of species distribution models more thoroughly

Species distribution models (SDMs) are empirical models relating species occurrence to environmental variables based on statistical or other response surfaces. SDMs can be used as a tool to solve some theoretical and applied ecological and environmental problems. The success of their applications depends on the accuracy of the models. In this study we propose an approach to thoroughly assess the accuracy of species distribution models. This includes three aspects: First is to use several accuracy indices that not only measure model discrimination capability, but also those that measure model reliability. The former is the power of the model that differentiates presences from absences; and the latter refers to the capability of the predicted probabilities to reflect the true probabilities that species occurs in individual locations. Previous studies have shown that some accuracy measures are sensitive to the prevalence of the test dataset, and that others are not. While all the reliability measures display this sensitivity to prevalence, only do some discriminatory measures fall into the latter group. Many researchers recommend the use of prevalence-insensitive measures in model accuracy assessment. However, using this approach the calibration power of the models cannot be assessed. We argue that calibration measures should also be provided in model accuracy assessments. The second aspect is to provide confidence intervals associated with the estimates of accuracy indices. Analytical methods, both parametric and nonparametric, have been introduced for constructing the confidence intervals for many accuracy indices. Computer-intensive methods (e.g. bootstrap and jackknife) can also be used to construct confidence intervals that are more attractive than the traditional analytical methods as (1) they have less statistical assumptions; and (2) they are virtually applicable to any accuracy measures. The third aspect is to provide an assessment of accuracy across a range of test data prevalence, since some accuracy indices are dependant on this quality of the test data. Test data with differing levels of prevalence will provide a range of results for the same accuracy index. Assessing the accuracy at only one level of prevalence will not provide a complete picture of the accuracy of the models. The range of test data prevalence can be set up by researchers according to their knowledge about the target species, or could be taken from the confidence interval of the population prevalence estimated from the sample data if the data can be considered as a random sample of the population. In this paper, we use an Australian native plant species, Forest Wire-grass (Tetrarrhena juncea), as an example to demonstrate our approach to more thoroughly assessing the accuracy of species distribution models. The accuracy of two models, one from a machine learning method (Random Forest, RF) and another from a statistical method (generalized additive model, GAM), were assessed using nine accuracy indices along a range of test data prevalence (i.e. the 95% confidence interval of the population prevalence estimated from the sample data using bootstrap percentile method), and a bootstrap method was used to construct the confidence intervals for the accuracy indices. With this approach, the species distribution models were thoroughly assessed.