Local-Global Fusion Network for Video Super-Resolution

被引:0
作者
Su, Dewei [1 ]
Wang, Hua [1 ]
Jin, Longcun [1 ]
Sun, Xianfang [2 ]
Peng, Xinyi [1 ]
机构
[1] South China Univ Technol, Sch Software Engn, Guangzhou 510006, Peoples R China
[2] Cardiff Univ, Sch Comp Sci & Informat, Cardiff CF24 3AA, Wales
基金
美国国家科学基金会;
关键词
Image resolution; Optical noise; Motion compensation; Estimation; Image reconstruction; Feature extraction; Optical network units; Convolutional neural networks; deep learning; improved deformable convolution; local-global feature fusion; video super-resolution; IMAGE QUALITY ASSESSMENT; SUPER-RESOLUTION;
D O I
10.1109/ACCESS.2020.3025780
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
The goal of video super-resolution technique is to address the problem of effectively restoring high-resolution (HR) videos from low-resolution (LR) ones. Previous methods commonly used optical flow to perform frame alignment and designed a framework from the perspective of space and time. However, inaccurate optical flow estimation may occur easily which leads to inferior restoration effects. In addition, how to effectively fuse the features of various video frames remains a challenging problem. In this paper, we propose a Local-Global Fusion Network (LGFN) to solve the above issues from a novel viewpoint. As an alternative to optical flow, deformable convolutions (DCs) with decreased multi-dilation convolution units (DMDCUs) are applied for efficient implicit alignment. Moreover, a structure with two branches, consisting of a Local Fusion Module (LFM) and a Global Fusion Module (GFM), is proposed to combine information from two different aspects. Specifically, LFM focuses on the relationship between adjacent frames and maintains the temporal consistency while GFM attempts to take advantage of all related features globally with a video shuffle strategy. Benefiting from our advanced network, experimental results on several datasets demonstrate that our LGFN can not only achieve comparative performance with state-of-the-art methods but also possess reliable ability on restoring a variety of video frames. The results on benchmark datasets of our LGFN are presented on https://github.com/BIOINSu/LGFN and the source code will be released as soon as the paper is accepted.
引用
收藏
页码:172443 / 172456
页数:14
相关论文
共 50 条
  • [31] Global Learnable Attention for Single Image Super-Resolution
    Su, Jian-Nan
    Gan, Min
    Chen, Guang-Yong
    Yin, Jia-Li
    Chen, C. L. Philip
    IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 2023, 45 (07) : 8453 - 8465
  • [32] Efficient lightweight network for video super-resolution
    Laigan Luo
    Benshun Yi
    Zhongyuan Wang
    Peng Yi
    Zheng He
    Neural Computing and Applications, 2024, 36 : 883 - 896
  • [33] Deep Unrolled Network for Video Super-Resolution
    Chiche, Benjamin Naoto
    Woiselle, Arnaud
    Frontera-Pons, Joana
    Starck, Jean-Luc
    2020 TENTH INTERNATIONAL CONFERENCE ON IMAGE PROCESSING THEORY, TOOLS AND APPLICATIONS (IPTA), 2020,
  • [34] Efficient lightweight network for video super-resolution
    Luo, Laigan
    Yi, Benshun
    Wang, Zhongyuan
    Yi, Peng
    He, Zheng
    NEURAL COMPUTING & APPLICATIONS, 2023, 36 (2) : 883 - 896
  • [35] Video super-resolution based on deep learning: a comprehensive survey
    Liu, Hongying
    Ruan, Zhubo
    Zhao, Peng
    Dong, Chao
    Shang, Fanhua
    Liu, Yuanyuan
    Yang, Linlin
    Timofte, Radu
    ARTIFICIAL INTELLIGENCE REVIEW, 2022, 55 (08) : 5981 - 6035
  • [36] Multi-Temporal Ultra Dense Memory Network for Video Super-Resolution
    Yi, Peng
    Wang, Zhongyuan
    Jiang, Kui
    Shao, Zhenfeng
    Ma, Jiayi
    IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, 2020, 30 (08) : 2503 - 2516
  • [37] Deformable 3D Convolution for Video Super-Resolution
    Ying, Xinyi
    Wang, Longguang
    Wang, Yingqian
    Sheng, Weidong
    An, Wei
    Guo, Yulan
    IEEE SIGNAL PROCESSING LETTERS, 2020, 27 : 1500 - 1504
  • [38] Video Super-Resolution With Convolutional Neural Networks
    Kappeler, Armin
    Yoo, Seunghwan
    Dai, Qiqin
    Katsaggelos, Aggelos K.
    IEEE TRANSACTIONS ON COMPUTATIONAL IMAGING, 2016, 2 (02) : 109 - 122
  • [39] Learning a Deep Dual Attention Network for Video Super-Resolution
    Li, Feng
    Bai, Huihui
    Zhao, Yao
    IEEE TRANSACTIONS ON IMAGE PROCESSING, 2020, 29 (29) : 4474 - 4488
  • [40] DDAN: A DEEP DUAL ATTENTION NETWORK FOR VIDEO SUPER-RESOLUTION
    Sun, Xiyue
    Li, Feng
    Bai, Huihui
    Zhao, Yao
    2021 IEEE INTERNATIONAL CONFERENCE ON MULTIMEDIA & EXPO WORKSHOPS (ICMEW), 2021,