An efficient mode decision algorithm for H.264/AVC intra prediction

Rate distortion optimization technique is adopted by H.264/AVC to select the best intra and inter prediction modes. It achieves remarkable improvement in compression performance, but the computational complexity of coding increases greatly. In order to reduce the computational complexity as much as possible while guaranteeing the video encoding quality and compression efficiency, this paper proposes a fast mode decision method based on the texture direction information of intra prediction modes and the encoding macroblocks. For intra luminance prediction, the proposed algorithm utilizes the smoothness of the encoding macroblock to select the suitable intra prediction block sizes, and then uses the texture direction difference to filter out low possibility prediction modes. The calculation expressions of texture direction difference can be derived by extracting texture direction features from intra prediction modes. For intra chrominance prediction, the candidate prediction modes are determined by a combination of texture direction difference and the sum of absolute transformed difference, which doesn’t significantly degrade peak-signal-noise-rate or increase bit rate. Based on the processing, the number of rate distortion cost calculations decreases dramatically, which indicates a significant reduction of computation cost for intra prediction. Compared with JM11.0 reference software, the proposed algorithm can cut down about 76.79 % total intra-frame coding time at the expense of only about 0.08 dB peak-signal-noise-rate degradation and 2.07 % bit rate increase. It proves that the proposed algorithm achieves a tradeoff between the rate distortion performance and the computational complexity.