Performance analysis of hierarchical transform coding with a large kernel for video codecs

In this study, the performance of hierarchical transform coding is analysed with design of an order-16 integer transform kernel. The proposed hierarchical transform-coding structure is constructed with a set of 4 x 4, 8 x 8 and 16 x 16 integer transforms of variable transform block sizes, which takes the advantages of both lower and higher transform kernels by flexibly adapting to varying image characteristics of video sequences with homogeneous and complex regions. The proposed hierarchical transform-coding structure is implemented as an extension to H.264/advanced video coding joint model. The authors show the effectiveness of the hierarchical variable-sized block transform scheme by analysing the quantisation effects and the correlation among neighbouring pixels in video sequences of different spatial resolutions. The experimental results show that: (i) the variable-sized block transform scheme with the hierarchical structure is advantageous to the texture regions with strong local edges and (ii) the higher-order-16 integer transform kernel itself is more effective for the homogeneous texture regions, which are often encountered in higher resolution sequences. Therefore these two features can complementarily work in an rate-distortion (RD) optimised manner for various characteristics of the input signals.