Learning Spatially Variant Linear Representation Models for Joint Filtering

被引:26
作者
Dong, Jiangxin [1 ]
Pan, Jinshan [1 ]
Ren, Jimmy S. [2 ,3 ]
Lin, Liang [4 ]
Tang, Jinhui [1 ]
Yang, Ming-Hsuan [5 ,6 ,7 ]
机构
[1] Nanjing Univ Sci & Technol, Sch Comp Sci & Engn, Nanjing 210094, Peoples R China
[2] SenseTime Res, Hong Kong, Peoples R China
[3] Shanghai Jiao Tong Univ, Qing Yuan Res Inst, Shanghai 200240, Peoples R China
[4] Sun Yat Sen Univ, Guangzhou 510275, Peoples R China
[5] Univ Calif Merced, Merced, CA 95344 USA
[6] Yonsei Univ, Seoul 03722, South Korea
[7] Google, Mountain View, CA 94043 USA
基金
国家重点研发计划; 美国国家科学基金会; 中国国家自然科学基金;
关键词
Task analysis; Convolutional neural networks; Image restoration; Computational modeling; Linear programming; Optimization; Kernel; Spatially variant linear representation model; convolutional neural network; joint filtering; DEPTH ENHANCEMENT; IMAGE;
D O I
10.1109/TPAMI.2021.3102575
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Joint filtering mainly uses an additional guidance image as a prior and transfers its structures to the target image in the filtering process. Different from existing approaches that rely on local linear models or hand-designed objective functions to extract the structural information from the guidance image, we propose a new joint filtering method based on a spatially variant linear representation model (SVLRM), where the target image is linearly represented by the guidance image. However, learning SVLRMs for vision tasks is a highly ill-posed problem. To estimate the spatially variant linear representation coefficients, we develop an effective approach based on a deep convolutional neural network (CNN). As such, the proposed deep CNN (constrained by the SVLRM) is able to model the structural information of both the guidance and input images. We show that the proposed approach can be effectively applied to a variety of applications, including depth/RGB image upsampling and restoration, flash deblurring, natural image denoising, and scale-aware filtering. In addition, we show that the linear representation model can be extended to high-order representation models (e.g., quadratic and cubic polynomial representations). Extensive experimental results demonstrate that the proposed method performs favorably against the state-of-the-art methods that have been specifically designed for each task.
引用
收藏
页码:8355 / 8370
页数:16
相关论文
共 79 条
[1]   A High-Quality Denoising Dataset for Smartphone Cameras [J].
Abdelhamed, Abdelrahman ;
Lin, Stephen ;
Brown, Michael S. .
2018 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 2018, :1692-1700
[2]  
Anwar S., 2020, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, P520
[3]   Real Image Denoising with Feature Attention [J].
Anwar, Saeed ;
Barnes, Nick .
2019 IEEE/CVF INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV 2019), 2019, :3155-3164
[4]   The Fast Bilateral Solver [J].
Barron, Jonathan T. ;
Poole, Ben .
COMPUTER VISION - ECCV 2016, PT III, 2016, 9907 :617-632
[5]  
Burger HC, 2012, PROC CVPR IEEE, P2392, DOI 10.1109/CVPR.2012.6247952
[6]  
Chang M., 2020, EUR C COMP VIS, P171
[7]  
Chen J, 2007, ACM T GRAPHIC, V26, DOI [10.1145/1239451.1239554, 10.1109/SARNOF.2007.4567317, 10.1145/1276377.1276506]
[8]   Fast Image Processing with Fully-Convolutional Networks [J].
Chen, Qifeng ;
Xu, Jia ;
Koltun, Vladlen .
2017 IEEE INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV), 2017, :2516-2525
[9]   Geodesic Image and Video Editing [J].
Criminisi, Antonio ;
Sharp, Toby ;
Rother, Carsten ;
Perez, Patrick .
ACM TRANSACTIONS ON GRAPHICS, 2010, 29 (05)
[10]   Image denoising by sparse 3-D transform-domain collaborative filtering [J].
Dabov, Kostadin ;
Foi, Alessandro ;
Katkovnik, Vladimir ;
Egiazarian, Karen .
IEEE TRANSACTIONS ON IMAGE PROCESSING, 2007, 16 (08) :2080-2095