Sparse Bayesian learning for off-grid DOA estimation with Gaussian mixture priors when both circular and non-circular sources coexist

In this paper, the problem of off-grid direction of arrival (DOA) estimation for the more general case of coexisting circular and non-circular signals is investigated from the perspective of sparse Bayesian learning (SBL). To utilize the second-order non-circularity of received signals, we carry out the DOA estimation by jointly representing the covariance and pseudo-covariance vectors. Although the sparse coefficient vectors of the covariance and pseudo-covariance vectors share common joint sparsity in the angular domain of non-circular sources, they have additional individual sparsity accounts for circular sources. Thus, the existing SBL methods based on joint sparsity will inevitably induce undesirable biases. To deal with this problem, a novel SBL method with the Gaussian mixture priors is developed. The proposed method can automatically identify the non-circular sources from the candidate angle grid and align the directional information of the non-circular sources in both the covariance and pseudo-covariance vectors. Moreover, the closed-form expressions for the perturbations of the covariance and pseudo-covariance vectors are also re-derived. Simulation results demonstrate that the proposed method achieves a significant performance improvement over existing methods. (C) 2019 Elsevier B.V. All rights reserved.