Encoding DCT coefficients in H.263 based on rate-distortion measurements

We propose a rate-distortion optimization algorithm which adjusts the magnitudes of the quantized DCT coefficients to achieve better video quality under a hit-rate constraint. The limited quantizer selections and the predetermined run-length code-word tables in the video standards are not optimal for all video sequences. Better performance can be achieved by adjusting the level of the quantized DCT coefficients, which minimizes the distortion subject to the rate constraint. After the DCT coefficients have been calculated, we select the quantization parameter (QP) and adjust the levels of the coefficients based on the rate-distortion measurement. The bit-streams thus generated by the algorithm conform to the standards. We present simulation results which show that the algorithm can achieve improvement compared to H.263+ TMN8, at the same bit-rate.