Hyperspectral endmember extraction is a process to estimate endmember signatures from the hyperspectral observations, in an attempt to study the underlying mineral composition of a landscape. However, estimating the number of endmembers, which is usually assumed to be known a priori in most endmember estimation algorithms (EEAs), still remains a challenging task. In this paper, assuming hyperspectral linear mixing model, we propose a hyperspectral data geometry-based approach for estimating the number of endmembers by utilizing successive endmember estimation strategy of an EEA. The approach is fulfilled by two novel algorithms, namely geometry-based estimation of number of endmembers-convex hull (GENE-CH) algorithm and affine hull (GENE-AH) algorithm. The GENE-CH and GENE-AH algorithms are based on the fact that all the observed pixel vectors lie in the convex hull and affine hull of the endmember signatures, respectively. The proposed GENE algorithms estimate the number of endmembers by using the Neyman-Pearson hypothesis testing over the endmember estimates provided by a successive EEA until the estimate of the number of endmembers is obtained. Since the estimation accuracies of the proposed GENE algorithms depend on the performance of the EEA used, a reliable, reproducible, and successive EEA, called p-norm-based pure pixel identification (TRI-P) algorithm is then proposed. The performance of the proposed TRI-P algorithm, and the estimation accuracies of the GENE algorithms are demonstrated through Monte Carlo simulations. Finally, the proposed GENE and TRI-P algorithms are applied to real AVIRIS hyperspectral data obtained over the Cuprite mining site, Nevada, and some conclusions and future directions are provided.
