Fast mode decision algorithm for HEVC intra coding based on texture partition and direction

High efficiency video coding (HEVC) is the newest video coding standard, which employs some advanced coding techniques as compared to the previous standard H.264. The flexible quad-tree partitioning of coding tree unit (CTU) and various candidate modes of prediction unit (PU) significantly promote the video compression efficiency; however, these techniques lead to a great amount of computational loads. In this paper, a fast mode decision algorithm for HEVC intra coding is proposed based on texture partition and direction. It consists of two sub-algorithms: the CTU depth range prediction (CDRP) and the intra-prediction mode selection (IPMS). The CDRP reduces the recursive partition number of coding unit (CU) based on the correlation between the CTU texture partition and the optimum CU partition, and it first calculates the texture partition flags of different-size CUs from bottom to top. Then, it employs these partition flags to predict the depth range of the current CTU and decide whether to terminate the CU partition in advance. In order to reduce the number of candidate PU modes for the Hadamard optimization, the IPMS first uses the three-step selection of the candidate modes. The first step selects the candidate modes based on the correlation between the texture directions and the optimum PU modes. The second step selects the candidate modes by using the best modes among the selected modes in the first step. The third step selects the candidate modes by using the spatial correlation of the optimum modes between the current PU and its adjacent PUs. Then, in order to reduce the number of candidate modes for the rate-distortion optimization, the IPMS utilizes the numerical relationship of the sorted Hadamard costs of above selected modes, the optimum modes of adjacent PUs and the statistical characteristics of the small-size PUs. Compared to the original algorithm in HEVC test model, the proposed overall algorithm can reduce 60% encoding time on average with only a 1.45% increase in Bjontegaard delta bit rate under the all-intra configuration. Compared to the most of state-of-the-art algorithms, the proposed overall algorithm has better computational performances and similar rate-distortion performances.