This paper presents a computer-aided polyp detection algorithm applicable to both colonoscopy and wireless capsule endoscopy (WCE). The proposed system has three integral parts: image enhancement, saliency map formation and Histogram of gradients (HOG) feature extraction for final classification. We propose a novel and efficient image enhancement algorithm, which enhances the saliency of clinically important features in endoscopic images. A saliency detection method is applied to the enhanced images to highlight the initial polyp candidates. In the classification stage, polyp candidates are selected after performing an image enhancement step and a saliency detection step. Exhaustive experiments have been performed on three publicly available databases: CVC ColonDB, CVC ClinicDB, and ETIS Larib to evaluate the performance of the proposed polyp detection algorithm. Comparison with the state-of-the-art methods shows that the proposed method outperforms the existing ones in terms of recall (=86.33%) and F2 score (=75.51%) for CVC ColonDB and in terms of recall (=74.04%) for the ETIS Larib dataset. With a significantly reduced number of search windows resulting from the saliency-based selection, the proposed scheme ensures a cost-effective and efficient polyp detection algorithm. (C) 2019 Elsevier Ltd. All rights reserved.