Multi-Frame Video Enhancement Using Virtual Frame Synthesized in Time Domain

被引：0
作者：
Ding D. [1 ]
Wu X. [1 ]
Tong J. [1 ]
Yao Z. [1 ]
Pan Z. [1 ,2 ]
机构：
[1] School of Information Science and Engineering, Hangzhou Normal University, Hangzhou
[2] Guangzhou NINED LLC, Guangzhou
来源：
Jisuanji Fuzhu Sheji Yu Tuxingxue Xuebao/Journal of Computer-Aided Design and Computer Graphics | 2020年 / 32卷 / 05期
关键词：
Convolutional neural network; Motion compensation; Video enhancement; Virtual frame;
D O I：
10.3724/SP.J.1089.2020.17952
中图分类号：
学科分类号：
摘要：
The convolutional neural network based video enhancement can effectively reduces compression artifacts, improving both the video coding efficiency and subjective quality. State-of-the-art methods usually adopt the single-frame enhancement strategies. However, video frames are also highly correlated in temporal domain, indicating that the reconstructed frames in temporal domain can also provide useful information to enhance the quality of current frame. To sufficiently utilize the temporal information, this paper proposes a spatial-temporal video enhancement method by introducing a virtual frame in time domain. We first employ an adaptive network to predict the virtual frame of current frame from its neighboring reconstructed frames. This virtual frame carries abundant temporal information. On the other hand, the current frame is also highly correlated in spatial domain. Hence we can combine spatial-temporal information together for extensive enhancement. To this end, we develop an enhancing network, which is structured in a progressive fusion manner, to combine both the virtual frame and the current frame for further frame fusion. Experimental results show that under random access configuration, the proposed method can obtain an average gain of 0.38 dB and 0.06 dB PSNR compared to the anchor H.265/HEVC and the single-frame-based strategy. Moreover, it outperforms the state-of-the-art multi-frame quality enhancement network (MFQE) 0.26 dB PSNR, whereas the number of parameters is only 12.2% of MFQE. The proposed method also significantly improves the subjective quality of the compressed videos. © 2020, Beijing China Science Journal Publishing Co. Ltd. All right reserved.
引用
收藏
页码：780 / 786
页数：6
相关论文
共 17 条
  • [11] He X Y, Hu Q, Zhang X T, Et al., Enhancing HEVC compressed videos with a partition-masked convolutional neural network, Proceedings of the 25th IEEE International Conference on Image Processing, pp. 216-220, (2018)
  • [12] Song X D, Yao J B, Zhou L L, Et al., A practical convolutional neural network as loop filter for intra frame, Proceedings of the IEEE International Conference on Image Processing, pp. 1133-1137, (2018)
  • [13] Mittal A, Moorthy A K, Bovik A C., No-reference image quality assessment in the spatial domain, IEEE Transactions on Image Processing, 21, 12, pp. 4695-4708, (2012)
  • [14] Niklaus S, Mai L, Liu F., Video frame interpolation via adaptive separable convolution, Proceedings of the IEEE International Conference on Computer Vision, pp. 261-270, (2017)
  • [15] Ilg E, Mayer N, Saikia T, Et al., FlowNet 2.0: evolution of optical flow estimation with deep networks, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2462-2470, (2017)
  • [16] He K M, Zhang X Y, Ren S Q, Et al., Deep residual learning for image recognition, Proceedings of the IEEE Conference on Conference on Computer Vision and Pattern Recognition, pp. 770-778, (2015)
  • [17] Kingma D P, Ba J., Adam: a method for stochastic optimization
12