Finite Samples and Uncertainty Estimates for Skill Measures for Seasonal Prediction

The expected value for various measures of skill for seasonal climate predictions is determined by the signal-to-noise ratio. The expected value, however, is only realized for long verification time series. In practice, the verifications for specific seasons-for example, forecasts for the December-February seasonal mean-seldom exceed a sample size of 30. The estimates of skill measure based on small verification time series, because of sampling errors, can have large departures from their expected value. An analysis of spread in the estimates of skill measures with the length of verification time series and for different signal-to-noise ratios is made. The analysis is based on the Monte Carlo approach and skill measures for deterministic, categorical, and probabilistic forecasts are considered. It is shown that the behavior of spread for various skill measures can be very different and it is not always the largest for the small values of signal-to-noise ratios.