A Bayesian Approach to Recreational Water Quality Model Validation and Comparison in the Presence of Measurement Error

Methods for measuring recreational water quality vary in analysis time, precision, availability, and cost. Decision-makers often use predictions from statistical models to compensate for the shortcomings of available measurements. However, model validation and comparison has largely omitted measurement error (defined here as variation in both sampling and the measurement technique) as an important source of uncertainty during validation. It is unknown how this omission affects estimates of model performance and comparisons between models. This study aims to fill this gap. First, we derive the bias incurred when omitting measurement error in calculating a model's mean squared error (MSE). We then develop a non-parametric validation method to correct estimates of MSE. To study other metrics of prediction performance (mean absolute error, sensitivity, precision, etc.) we develop a second validation method that uses simulations from a Bayesian validation model. These methods are applied to a comparison of two prediction models (random forest and nearest neighbor) used to predict the level of fecal indicator bacteria at nine recreational beaches in the city of Chicago. We find that accounting for measurement error significantly changes estimates of model performance. Moreover, it reveals substantial uncertainty underlying some of these estimates.