Fast intra-mode decision in H.264 video coding using simple directional masks

The H.264/AVC standard improves coding efficiency significantly compared with existing standards. This is at the expense of great computational complexity at the encoder. H.264/AVC adopts rate-distortion optimization (RDO) technique to obtain the best result while maximizing visual quality and minimizing bitrates. To carry out RDO, the encoder undergoes all possible combinations of coding modes exhaustively. H.264/AVC exploits directional spatial correlation with adjacent blocks for intra prediction. There are 9 directional modes for 4x4 luma blocks, 4 directional modes for 16x16 macroblocks and 8x8 chroma. blocks, respectively. To determine the appropriate direction, based on the RDO computation, the number of mode combinations in each MB is 592. For a fast intra mode decision, we propose an algorithm to reduce the number of RDO computations using simple directional masks and adjacent mode information. The proposed method carries out 4 RDO calculations at the most for 4x4 luma blocks, and 2 RDO computations for 16x16 luma. blocks and 8x8 chroma blocks, respectively. We verified the proposed algorithm by implementing on JM reference software and comparing with the case of exhaustive search. Simulation results show the proposed method reduces the ending time up to 70% with negligible loss of PSNR and bitrate increase.