Covariance Matrix Estimation with Multi-Regularization Parameters based on MDL Principle

Regularization is a solution for the problem of unstable estimation of covariance matrix with a small sample set in Gaussian classifier. In many applications such as image restoration, sparse representation, we have to deal with multi-regularization parameters problem. In this paper, the case of covariance matrix estimation with multi-regularization parameters is investigated, and an estimate method called as KLIM_L is derived theoretically based on Minimum Description Length (MDL) principle for the small sample size problem with high dimension setting. KLIM_L estimator can be regarded as a generalization of KLIM estimator in which local difference in each dimension is considered. Under the framework of MDL principle, a selection method of multi-regularization parameters is also developed based on the minimization of the Kullback-Leibler information measure, which is simply and directly estimated by point estimation under the approximation of two-order Taylor expansion. The computational cost to estimate multi-regularization parameters with KLIM_L method is less than those with RDA (Regularized Discriminant Analysis) and LOOC (leave-one-out covariance matrix estimate) in which cross validation technique is adopted. Experiments show that higher classification accuracy can be achieved by using the proposed KLIM_L estimator.