A rate-distortion optimization algorithm for rate control in H.264

This paper presents a novel rate-distortion (R-D) Joint optimization rate control (RC) algorithm for H.264 encoding. For RC in H.264, one of the most important topics is to model the R-D characteristics accurately. To achieve this, an efficient linear model is proposed to model the distortion-quantization (D-Q) relation. With the proposed linear D-Q model, a closed-form solution is derived to calculate the optimal quantization parameter for encoding each macroblock (MB). The proposed RC algorithm can be applied to both P-frames and B-frames. It is shown by experimental results that the proposed algorithm can control the bit rates accuratly with the R-D performance better than that of the RC algorithm JVT-G012 implemented in the H.264 reference software JM9.5.