Fast mode decision for intra prediction in H.264/AVC encoder

The H.264/AVC video coding standard uses the rate distortion optimization (RDO) method to improve the compression performance in the intra prediction. Whereas the computational complexity is increased comparing with previous standards due to this method, even though this standard selects the best coding mode for the current macroblock. In this paper, a fast intra mode decision algorithm for H.264/AVC encoder based on dominant edge direction (DED) is proposed. The algorithm uses the approximation of discrete cosine transform (DCT) coefficient formula. By detecting the DED before intra prediction, 3 modes instead of 9 modes are chosen for RDO calculation to decide the best mode in the 4x4 luma block. For the 16x16 luma and the 8x8 chroma, instead of 4 modes, only 2 modes are chosen. Experimental results show that the computation time of the proposed algorithm is decreased to about 71% of the full search method in the reference code with negligible quality loss.