Rain Removal From Light Field Images With 4D Convolution and Multi-Scale Gaussian Process

被引:10
作者
Yan, Tao [1 ]
Li, Mingyue [1 ]
Li, Bin [1 ]
Yang, Yang [2 ]
Lau, Rynson W. H. [3 ]
机构
[1] Jiangnan Univ, Sch Artificial Intelligence & Comp Sci, Wuxi 214122, Peoples R China
[2] Jiangsu Univ, Dept Comp Sci, Zhenjiang 212013, Peoples R China
[3] City Univ Hong Kong, Dept Comp Sci, Hong Kong, Peoples R China
基金
中国国家自然科学基金;
关键词
Rain; Semisupervised learning; Training; Image restoration; Feature extraction; Task analysis; Estimation; Light field images; rain removal; 4D convolution; semi-supervised learning; Gaussian process; STREAK REMOVAL; SINGLE; NETWORK; MODEL;
D O I
10.1109/TIP.2023.3234692
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Existing deraining methods focus mainly on a single input image. However, with just a single input image, it is extremely difficult to accurately detect and remove rain streaks, in order to restore a rain-free image. In contrast, a light field image (LFI) embeds abundant 3D structure and texture information of the target scene by recording the direction and position of each incident ray via a plenoptic camera. LFIs are becoming popular in the computer vision and graphics communities. However, making full use of the abundant information available from LFIs, such as 2D array of sub-views and the disparity map of each sub-view, for effective rain removal is still a challenging problem. In this paper, we propose a novel method, 4D-MGP-SRRNet, for rain streak removal from LFIs. Our method takes as input all sub-views of a rainy LFI. To make full use of the LFI, it adopts 4D convolutional layers to simultaneously process all sub-views of the LFI. In the pipeline, the rain detection network, MGPDNet, with a novel Multi-scale Self-guided Gaussian Process (MSGP) module is proposed to detect high-resolution rain streaks from all sub-views of the input LFI at multi-scales. Semi-supervised learning is introduced for MSGP to accurately detect rain streaks by training on both virtual-world rainy LFIs and real-world rainy LFIs at multi-scales via computing pseudo ground truths for real-world rain streaks. We then feed all sub-views subtracting the predicted rain streaks into a 4D convolution-based Depth Estimation Residual Network (DERNet) to estimate the depth maps, which are later converted into fog maps. Finally, all sub-views concatenated with the corresponding rain streaks and fog maps are fed into a powerful rainy LFI restoring model based on the adversarial recurrent neural network to progressively eliminate rain streaks and recover the rain-free LFI. Extensive quantitative and qualitative evaluations conducted on both synthetic LFIs and real-world LFIs demonstrate the effectiveness of our proposed method.
引用
收藏
页码:921 / 936
页数:16
相关论文
共 76 条
  • [1] EAGNet: Elementwise Attentive Gating Network-Based Single Image De-Raining With Rain Simplification
    Ahn, Namhyun
    Jo, So Yeon
    Kang, Suk-Ju
    [J]. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, 2022, 32 (02) : 608 - 620
  • [2] [Anonymous], 2010, P 27 INT C MACH LEAR
  • [3] [Anonymous], STANF LYTR LIGHT FIE
  • [4] Learning A Cascaded Non-Local Residual Network for Super-resolving Blurry Images
    Bai, Haoran
    Cheng, Songsheng
    Tang, Jinhui
    Pan, Jinshan
    [J]. 2021 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION WORKSHOPS, CVPRW 2021, 2021, : 223 - 232
  • [5] Blender, ABOUT US
  • [6] Pre-Trained Image Processing Transformer
    Chen, Hanting
    Wang, Yunhe
    Guo, Tianyu
    Xu, Chang
    Deng, Yiping
    Liu, Zhenhua
    Ma, Siwei
    Xu, Chunjing
    Xu, Chao
    Gao, Wen
    [J]. 2021 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, CVPR 2021, 2021, : 12294 - 12305
  • [7] A Generalized Low-Rank Appearance Model for Spatio-Temporally Correlated Rain Streaks
    Chen, Yi-Lei
    Hsu, Chiou-Ting
    [J]. 2013 IEEE INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV), 2013, : 1968 - 1975
  • [8] Demir U, 2018, Arxiv, DOI [arXiv:1803.07422, DOI 10.48550/ARXIV.1803.07422]
  • [9] Rain Streak Removal From Light Field Images
    Ding, Yuyang
    Li, Mingyue
    Yan, Tao
    Zhang, Fan
    Liu, Yuan
    Lau, Rynson W. H.
    [J]. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, 2022, 32 (02) : 467 - 482
  • [10] Successive Graph Convolutional Network for Image De-raining
    Fu, Xueyang
    Qi, Qi
    Zha, Zheng-Jun
    Ding, Xinghao
    Wu, Feng
    Paisley, John
    [J]. INTERNATIONAL JOURNAL OF COMPUTER VISION, 2021, 129 (05) : 1691 - 1711