Deep Convolution Modulation for Image Super-Resolution

被引:12
作者
Huang, Yuanfei [1 ]
Li, Jie [2 ]
Hu, Yanting [3 ]
Huang, Hua [1 ]
Gao, Xinbo [4 ]
机构
[1] Beijing Normal Univ, Sch Artificial Intelligence, Beijing 100875, Peoples R China
[2] Xidian Univ, Sch Elect Engn, Video & Image Proc Syst Lab, Xian 710071, Peoples R China
[3] Xinjiang Med Univ, Sch Med Engn & Technol, Urumqi 830011, Peoples R China
[4] Chongqing Univ Posts & Telecommun, Chongqing Key Lab Image Cognit, Chongqing 400065, Peoples R China
基金
中国国家自然科学基金;
关键词
Convolution; Modulation; Computer architecture; Task analysis; Superresolution; Kernel; Training; Image super-resolution; convolutional neural networks; convolution modulation; NETWORK; ATTENTION; INTERPOLATION;
D O I
10.1109/TCSVT.2023.3317486
中图分类号
TM [电工技术]; TN [电子技术、通信技术];
学科分类号
0808 ; 0809 ;
摘要
Recently, deep-learning-based super-resolution methods have achieved excellent performances, but mainly focus on training a single generalized deep network by feeding numerous samples. Yet intuitively, each image has its specific representation, and is expected to acquire an adaptive model. For this issue, we propose a novel convolution modulation (CoMo) mechanism to build image-specific deep networks, by exploiting the principal information of the feature to generate a modulation weight, and thereby adaptively modulating the kernel weights of convolution without any additional parameters, which outperforms the vanilla convolution and several existing attention mechanisms when embedding into the state-of-the-art architectures. To optimize the modulated convolutions in mini-batch training, we introduce an image-specific optimization (IsO) algorithm, which tackles the infeasibility of the conventional optimization algorithms on this issue. Furthermore, we investigate the effect of CoMo on state-of-the-art architectures and design a new CoMoNet architecture by employing the U-style residual learning and hourglass dense block learning, which is an appropriate architecture to utmost improve the effectiveness of CoMo theoretically. Extensive experiments on benchmarks show that the proposed methods achieve superior performances and higher flexibility against the state-of-the-art SISR and blind SR methods. The code is available at github.com/YuanfeiHuang/CoMoNet.
引用
收藏
页码:3647 / 3662
页数:16
相关论文
共 90 条
[1]   Fast, Accurate, and Lightweight Super-Resolution with Cascading Residual Network [J].
Ahn, Namhyuk ;
Kang, Byungkon ;
Sohn, Kyung-Ah .
COMPUTER VISION - ECCV 2018, PT X, 2018, 11214 :256-272
[2]  
[Anonymous], Multiscale context aggregation by dilated convolutions
[3]   Densely Residual Laplacian Super-Resolution [J].
Anwar, Saeed ;
Barnes, Nick .
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 2022, 44 (03) :1192-1204
[4]   Contour Detection and Hierarchical Image Segmentation [J].
Arbelaez, Pablo ;
Maire, Michael ;
Fowlkes, Charless ;
Malik, Jitendra .
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 2011, 33 (05) :898-916
[5]   Single Image Super-Resolution via a Holistic Attention Network [J].
Niu, Ben ;
Wen, Weilei ;
Ren, Wenqi ;
Zhang, Xiangde ;
Yang, Lianping ;
Wang, Shuzhen ;
Zhang, Kaihao ;
Cao, Xiaochun ;
Shen, Haifeng .
COMPUTER VISION - ECCV 2020, PT XII, 2020, 12357 :191-207
[6]   Low-Complexity Single-Image Super-Resolution based on Nonnegative Neighbor Embedding [J].
Bevilacqua, Marco ;
Roumy, Aline ;
Guillemot, Christine ;
Morel, Marie-Line Alberi .
PROCEEDINGS OF THE BRITISH MACHINE VISION CONFERENCE 2012, 2012,
[7]   SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning [J].
Chen, Long ;
Zhang, Hanwang ;
Xiao, Jun ;
Nie, Liqiang ;
Shao, Jian ;
Liu, Wei ;
Chua, Tat-Seng .
30TH IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2017), 2017, :6298-6306
[8]   Accelerating the Super-Resolution Convolutional Neural Network [J].
Dong, Chao ;
Loy, Chen Change ;
Tang, Xiaoou .
COMPUTER VISION - ECCV 2016, PT II, 2016, 9906 :391-407
[9]   Image Super-Resolution Using Deep Convolutional Networks [J].
Dong, Chao ;
Loy, Chen Change ;
He, Kaiming ;
Tang, Xiaoou .
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 2016, 38 (02) :295-307
[10]   Image Deblurring and Super-Resolution by Adaptive Sparse Domain Selection and Adaptive Regularization [J].
Dong, Weisheng ;
Zhang, Lei ;
Shi, Guangming ;
Wu, Xiaolin .
IEEE TRANSACTIONS ON IMAGE PROCESSING, 2011, 20 (07) :1838-1857