Effective subblock-based and pixel-based fast direction detections for H.264 intra prediction

In H.264/AVC intra-frame coding, the rate-distortion optimization (RDO) is employed to select the optimal coding mode to achieve the minimum rate-distortion cost. Due to a large number of combinations of coding modes, the computational burden becomes extremely high in intra-prediction. In this paper, we propose two fast, efficient but reliable direction detection algorithms by computing subblock and pixel direction differences. Both proposed methods effectively estimate the edge direction inside the block to narrow down the predictive modes to reduce the RDO computation. Experimental results show that the proposed methods can reduce the encoding time by about 60% with negligible loss of coding performance. For hardware realization, a fast mode decision VLSI circuit for intra-prediction with the silicon core size of 0.12 × 0.12 mm2 at 0.18-μm CMOS technology is implemented. The fast mode decision VLSI in three-stage pipelined architecture operated at 173 MHz can encode 30 fps real-time videos up to Level 5. © 2006 IEEE.