A debiased ranked probability skill score to evaluate probabilistic ensemble forecasts with small ensemble sizes

被引:77
作者
Müller, WA
Appenzeller, C
Doblas-Reyes, FJ
Liniger, MA
机构
[1] Swiss Fed Off Meterol & Climatol MeteoSwiss, Zurich, Switzerland
[2] European Ctr Medium Range Weather Forecasts, Reading RG2 9AX, Berks, England
关键词
D O I
10.1175/JCLI3361.1
中图分类号
P4 [大气科学(气象学)];
学科分类号
0706 ; 070601 ;
摘要
The ranked probability skill score (RPSS) is a widely used measure to quantify the skill of ensemble forecasts. The underlying score is defined by the quadratic norm and is comparable to the mean squared error (mse) but it is applied in probability space. It is sensitive to the shape and the shift of the predicted probability distributions. However, the RPSS shows a negative bias for ensemble systems with small ensemble size, as recently shown. Here, two strategies are explored to tackle this flaw of the RPSS. First, the RPSS is examined for different norms L (RPSSL). It is shown that the RPSSL=1 based on the absolute rather than the squared difference between forecasted and observed cumulative probability distribution is unbiased; RPSSL defined with higher-order norms show a negative bias. However, the RPSSL=1 is not strictly proper in a statistical sense. A second approach is then investigated, which is based on the quadratic norm but with sampling errors in climatological probabilities considered in the reference forecasts. This technique is based on strictly proper scores and results in an unbiased skill score, which is denoted as the debiased ranked probability skill score (RPSSD) hereafter. Both newly defined skill scores are independent of the ensemble size, whereas the associated confidence intervals are a function of the ensemble size and the number of forecasts. The RPSSL-1 and the RPSSD are then applied to the winter mean [December-January-February (DJF)] near-surface temperature predictions of the ECMWF Seasonal Forecast System 2. The overall structures of the RPSSL=1 and the RPSSD are more consistent and largely independent of the ensemble size, unlike the RPSSL-2. Furthermore, the minimum ensemble size required to predict a climate anomaly given a known signal-to-noise ratio is determined by employing the new skill scores. For a hypothetical setup comparable to the ECMWF hindcast system (40 members and 15 hindcast years), statistically significant skill scores were only found for a signal-to-noise ratio larger than similar to 0.3.
引用
收藏
页码:1513 / 1523
页数:11
相关论文
共 50 条
[31]   A skill score based on economic value for probability forecasts [J].
Wilks, DS .
METEOROLOGICAL APPLICATIONS, 2001, 8 (02) :209-219
[32]   On the effect of ensemble size on the discrete and continuous ranked probability scores [J].
Ferro, Christopher A. T. ;
Richardson, David S. ;
Weigel, Andreas P. .
METEOROLOGICAL APPLICATIONS, 2008, 15 (01) :19-24
[33]   Observational probability method to assess ensemble precipitation forecasts [J].
Santos, Carlos ;
Ghelli, Anna .
QUARTERLY JOURNAL OF THE ROYAL METEOROLOGICAL SOCIETY, 2012, 138 (662) :209-221
[34]   Skill prediction of local weather forecasts based on the ECMWF ensemble [J].
Ziehmann, C .
NONLINEAR PROCESSES IN GEOPHYSICS, 2001, 8 (06) :419-428
[35]   Ensemble-based ozone forecasts: Skill and economic value [J].
Pagowski, Mariusz ;
Grell, Georg A. .
JOURNAL OF GEOPHYSICAL RESEARCH-ATMOSPHERES, 2006, 111 (D23)
[36]   Decomposition of a New Proper Score for Verification of Ensemble Forecasts [J].
Christensen, H. M. .
MONTHLY WEATHER REVIEW, 2015, 143 (05) :1517-1532
[37]   The impact of horizontal resolution and ensemble size on probabilistic forecasts of precipitation by the ECMWF Ensemble Prediction System [J].
Mullen, SL ;
Buizza, R .
WEATHER AND FORECASTING, 2002, 17 (02) :173-191
[38]   Sequential aggregation of probabilistic forecasts-Application to wind speed ensemble forecasts [J].
Zamo, Michael ;
Bel, Liliane ;
Mestre, Olivier .
JOURNAL OF THE ROYAL STATISTICAL SOCIETY SERIES C-APPLIED STATISTICS, 2021, 70 (01) :202-225
[39]   The reduction continuous rank probability score for evaluating discharge forecasts from hydrological ensemble prediction systems [J].
Trinh, B. N. ;
Thielen-del Pozo, J. ;
Thirel, G. .
ATMOSPHERIC SCIENCE LETTERS, 2013, 14 (02) :61-65
[40]   Calibration of probabilistic, ensemble precipitation forecasts by an artificial neural network [J].
Mullen, SL ;
Poulton, MM ;
Buizza, R .
SECOND CONFERENCE ON ARTIFICIAL INTELLIGENCE, 2000, :100-101