Fast rate-distortion optimized coding mode decision for H.264

n H.264/MPEG-4 AVC a wide range of different prediction block forms and prediction signal generation methods are available. By selecting an optimal combination of coding modes from the multiple possible combinations of such modes, improvements in coding efficiency can be achieved. However, when using the rate-distortion optimized coding mode decision method based on the Lagrange multipliers, at the same time as seeing a significant improvement in coding efficiency, we are faced with the problem that the mode decision procedure becomes extremely demanding computationally. The H.264/MPEG-4 AVC high profile introduces adaptive block size transformations thereby making the number of combinations of coding mode that can be selected even larger than under the main profile. In this paper we investigate a method for hierarchically and adaptively reducing the number of mode combinations. Specifically we propose a method for quickly deciding the coding mode while limiting the reductions in coding efficiency by the correlation information between two mode decision cost functions in accordance with a quantization parameter. The results of experiments confirm that by using the proposed method, the encoding time excluding the motion search can be reduced by up to 4 times for the main profile and by up to 7 times for the high profile as compared to rate-distortion optimized coding mode decision. (C) 2007 Wiley Periodicals, Inc. Electron Comm Jpn Pt 3, 90(9): 41-55, 2007; Published online in Wiley InterScience (www.interscience.wiley.com).