In this paper, an effective directional interpolation- and inpainting-based impulse noise removal algorithm is proposed. Firstly, each noisy pixel is classified to either the low-density noise or the middle/high density noise. Secondly, a directional interpolation-based noise removal procedure is proposed to denoise the low-density noise. Thirdly, an inpainting-based noise removal procedure is proposed to denoise the middle/high-density noise. Based on ten typical test images, each image with noise level ranging from 30 to 90%, the experimental results demonstrate that in terms of peak-signal-to-noise-ratio (PSNR), structural similarity index (SSIM), and visual effect, the proposed algorithm has the best quality performance when compared with six state-of-the-art noise removal algorithms.