Progressive Motion Boosting for Video Frame Interpolation

被引:2
作者
Xiao, Jing [1 ]
Xu, Kangmin [1 ]
Hu, Mengshun [1 ]
Liao, Liang [2 ]
Wang, Zheng [1 ]
Lin, Chia-Wen [3 ,4 ]
Wang, Mi [5 ]
Satoh, Shin'ichi [6 ]
机构
[1] Wuhan Univ, Sch Comp Sci, Wuhan 430072, Peoples R China
[2] Nanyang Technol Univ, Sch Comp Sci & Engn, Singapore 639798, Singapore
[3] Natl Tsing Hua Univ, Dept Elect Engn, Hsinchu 30013, Taiwan
[4] Natl Tsing Hua Univ, Inst Commun Engn, Hsinchu 30013, Taiwan
[5] Wuhan Univ, State Key Lab Informat Engn Surveying Mapping & Re, Wuhan 430072, Peoples R China
[6] Natl Inst Informat, Digital Content & Media Sci Res Div, Tokyo 1018430, Japan
基金
中国国家自然科学基金;
关键词
Frame interpolation; motion estimation; multi-scale framework; progressive boosting;
D O I
10.1109/TMM.2022.3233310
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
Video frame interpolation has made great progress in estimating advanced optical flow and synthesizing in-between frames sequentially. However, frame interpolation involving various resolutions and motions remains challenging due to limited or fixed pre-trained networks. Inspired by the success of the coarse-to-fine scheme for video frame interpolation, i.e., gradually interpolating frames of different resolutions, we propose a progressive boosting network (ProBoost-Net) based on a multi-scale framework to achieve flexible recurrent scales and then gradually optimize optical flow estimation and frame interpolation. Specifically, we designed a dense motion boosting (DMB) module to transfer features close to real motion to the decoded features from the later scales, which provides complementary information to refine the motion further. Furthermore, to ensure the accuracy of the estimated motion features at each scale, we propose a motion adaptive fusion (MAF) module that adaptively deals with motions with different receptive fields according to the motion conditions. Thanks to the framework's flexible recurrent scales, we can customize the number of scales and make trade-offs between computation and quality depending on the application scenario. Extensive experiments with various datasets demonstrated the superiority of our proposed method over state-of-the-art approaches in various scenarios.
引用
收藏
页码:8076 / 8090
页数:15
相关论文
共 74 条
[61]  
van Amersfoort J, 2019, Arxiv, DOI arXiv:1711.06045
[62]  
Waltl M., 2012, P 20 ACM INT C MULT, P1469, DOI [10.1145/2393347.2396522, DOI 10.1145/2393347.2396522]
[63]   Non-local Neural Networks [J].
Wang, Xiaolong ;
Girshick, Ross ;
Gupta, Abhinav ;
He, Kaiming .
2018 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 2018, :7794-7803
[64]   Frame Rate Conversion Based High Efficient Compression Method for Video Satellite [J].
Wang, Xu ;
Hu, Ruimin ;
Xiao, Jing .
ADVANCES IN MULTIMEDIA INFORMATION PROCESSING - PCM 2018, PT II, 2018, 11165 :35-44
[65]  
Wu HN, 2023, Arxiv, DOI arXiv:2211.04894
[66]   FAST-VQA: Efficient End-to-End Video Quality Assessment with Fragment Sampling [J].
Wu, Haoning ;
Chen, Chaofeng ;
Hou, Jingwen ;
Liao, Liang ;
Wang, Annan ;
Sun, Wenxiu ;
Yan, Qiong ;
Lin, Weisi .
COMPUTER VISION - ECCV 2022, PT VI, 2022, 13666 :538-554
[67]  
Wu HN, 2022, Arxiv, DOI arXiv:2206.09853
[68]   Optimizing Video Prediction via Video Frame Interpolation [J].
Wu, Yue ;
Wen, Qiang ;
Chen, Qifeng .
2022 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2022), 2022, :17793-17802
[69]  
Xinghao Chen, 2020, Computer Vision - ECCV 2020. 16th European Conference. Proceedings. Lecture Notes in Computer Science (LNCS 12351), P614, DOI 10.1007/978-3-030-58539-6_37
[70]   Video Enhancement with Task-Oriented Flow [J].
Xue, Tianfan ;
Chen, Baian ;
Wu, Jiajun ;
Wei, Donglai ;
Freeman, William T. .
INTERNATIONAL JOURNAL OF COMPUTER VISION, 2019, 127 (08) :1106-1125