Bayesian estimation of the observation-error covariance matrix in ensemble-based filters

We develop a Bayesian technique for estimating the parameters in the observation-noise covariance matrix R-t for ensemble data assimilation. We design a posterior distribution by using the ensemble-approximated likelihood and a Wishart prior distribution and present an iterative algorithm for parameter estimation. The temporal smoothness of R-t can be controlled by an adequate choice of two parameters of the prior distribution, the covariance matrix S and the number of degrees of freedom . The parameter can be estimated by maximizing the marginal likelihood. The present formalism can handle cases in which the number of data points or data positions varies with time, the former of which is exemplified in the experiments. We present an application to a coupled atmosphere-ocean model under each of the following assumptions: R-t is a scalar multiple of a fixed matrix (R-t=(t)sigma, where (t) is the scalar parameter and sigma is the fixed matrix), R-t is diagonal, R-t has fixed eigenvectors or R-t has no specific structure. We verify that the proposed algorithm works well and that only a limited number of iterations are necessary. When R-t has one of the structures mentioned above, by assuming S to be the previous estimate we obtain a Bayesian estimate of R-t that varies smoothly in time compared with the maximum-likelihood estimate. When R-t has no specific structure, we need to regularize S to maintain the positive-definiteness. Through twin experiments, we find that the best estimate of R-t is, in general, obtained by a combination of structure-free R-t and tapered S using decorrelation lengths of half the size of the model ocean basin. From experiments using real observations, we find that the estimates of the structured R-t lead to overfitting of the data compared with the structure-free R-t.