Detection-Estimation of Very Close Emitters: Performance Breakdown, Ambiguity, and General Statistical Analysis of Maximum-Likelihood Estimation

We reexamine the well-known problem of "threshold behavior" or "performance breakdown" in the detection-estimation of very closely spaced emitters. In this extreme regime, we analyze the performance for maximum-likelihood estimation (MLE) of directions-of-arrival (DOA) for two close Gaussian sources over the range of sample volumes and signal-to-noise ratios (SNRs) where the correct number of sources is reliably estimated by information-theoretic criteria (ITC), but where one of the DOA estimates is severely erroneous ("outlier"). We show that random matrix theory (RMT) applied to the evaluation of theoretical MLE performance gives a relatively simple and accurate analytical description of the threshold behavior of MLE and ITC. In particular, the introduced "single-cluster" criterion provides accurate "ambiguity bounds" for the outliers.