Rate-distortion optimization for mode decision with sequence statistics in H.264/AVC

There are 21 possible candidate macroblock modes in H.264/AVC to encode a macroblock of any sequences using Rate-Distortion Optimization (RDO) mode decision method. RDO is one of the tools in H.264 to choose the best macroblock mode having minimum cost function which is consisted of transformed, quantized macroblock coefficients and Lagrange multiplier. Lagrange multiplier for macroblock mode decision was derived from the assumption of high-rate model approach, and it's a function of constant value 0.85 and QP. Since the use of constant value 0.85 cannot be suitable to any kinds of sequences, especially, in a low-rate environment, it can be degraded the performance of encoder. In these reason, our paper proposed to improve the coding efficiency by controlling Lagrange multiplier adaptively according to sequence statistics, or variance of residual image, in RDO mode decision.