Source number estimation based on sequential ratio for the invariant subspace of matrices

In this paper, we propose a method to estimate the number of sources based on the sequential ratio for the invariant subspace of matrices that can provide accurate results in scenarios involving colored noise and a small number of snapshots. We first form a pair of projection matrices on each eigenvector of the sample covariance matrices by using the invariant subspace of matrices. We then use the generalized cosine-squared method to evaluate the differences of M pairs of projection matrices, according to the gap between the projection matrices corresponding to the eigenvectors of the signals and noise. This yields a distinct transition from the signal subspace to the noise subspace. Based on this, we develop a sequential ratio-based method to precisely determine the number of sources by identifying the position at which the ratio of differences significantly decreases. The proposed method can identify the index value at which a significant drop occurs in the number of sources according to the transition from the signal subspace to the noise subspace. The simulation results showed that the proposed method is superior to the current methods, and excels particularly in conditions involving colored noise.