On sample eigenvalues in a generalized spiked population model

In the spiked population model introduced by Johnstone (2001) [11], the population covariance matrix has all its eigenvalues equal to unit except for a few fixed eigenvalues (spikes). The question is to quantify the effect of the perturbation caused by the spike eigenvalues. Baik and Silverstein (2006) [5] establishes the almost sure limits of the extreme sample eigenvalues associated to the spike eigenvalues when the population and the sample sizes become large. In a recent work Bai and Yao (2008) [4], we have provided the limiting distributions for these extreme sample eigenvalues. In this paper, we extend this theory to a generalized spiked population model where the base population covariance matrix is arbitrary, instead of the identity matrix as in Johnstone's case. As the limiting spectral distribution is arbitrary here, new mathematical tools, different from those in Baik and Silverstein (2006) [5], are introduced for establishing the almost sure convergence of the sample eigenvalues generated by the spikes. (C) 2011 Elsevier Inc. All rights reserved.