This paper focuses on the spectral unmixing technique for analyzing hyperspectral image (HSI). In this paper, we first prove that the reconstruction errors and the abundance anomalies (AAs, abundances that are negative or greater than one) are effective in measuring the purity of pixels. Then, due to the continuity of the objects in the space, the endmembers are assumed to be located at some noticeable areas in residual and AA maps. A saliency-based endmember detection (SED) algorithm which aims at iteratively extracting endmembers from the residual and AA maps is proposed, where the visual attention mechanism is developed to understand and analyze the spatial pattern of endmembers. In addition, when searching for new endmembers, the spectral properties are also utilized to promote the robustness of the proposed method. The experimental results on both simulated data and real hyperspectral data illustrate the merits and viability of the proposed algorithm.