In general, image restoration problems are ill-posed and need to be regularized. For applications such as real-time video, fast restorations are also needed to keep up with the frame rate. Restoration based on 2-D FFT's provides a fast implementation assuming a constant regularization term over the image. Unfortunately, this assumption creates significant ringing artifacts on edges as well as blurrier edges in the restored image. On the other hand, shift-variant regularization will reduce edge artifacts and provide better quality but it destroys the structure that makes use of the 2-D FFT possible, thus no longer have the computational efficiency of the FFT. In this paper, we use a Bayesian approach-maximum a posterior (MAP) estimation to compute an estimate of the original image given the blurred image. To avoid the smoothing of edges, shift-variant regularization must be used. The Huber-Markov random field model is applied to preserve the discontinuities on edges. For fast minimization of the above model, a new algorithm involving the Sherman-Morrison matrix inversion lemma is proposed. This results in a restored image with good edge preservation and less computation. Experiments show restored images with sharper edges. Convergence is fast, and the computational speed can be improved considerably by breaking the image into sub-images.