Learning Adaptive Warping for Real-World Rolling Shutter Correction

被引:15
作者
Cao, Mingdeng [1 ]
Zhong, Zhihang [2 ]
Wang, Jiahao [1 ]
Zheng, Yinqiang [2 ]
Yang, Yujiu [1 ]
机构
[1] Tsinghua Univ, Tsinghua Shenzhen Int Grad Sch, Beijing, Peoples R China
[2] Univ Tokyo, Tokyo, Japan
来源
2022 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2022) | 2022年
基金
中国国家自然科学基金;
关键词
D O I
10.1109/CVPR52688.2022.01726
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
This paper proposes the first real-world rolling shutter (RS) correction dataset, BS-RSC, and a corresponding model to correct the RS frames in a distorted video. Mobile devices in the consumer market with CMOS-based sensors for video capture often result in rolling shutter effects when relative movements occur during the video acquisition process, calling for RS effect removal techniques. However, current state-of-the-art RS correction methods often fail to remove RS effects in real scenarios since the motions are various and hard to model. To address this issue, we propose a real-world RS correction dataset BS-RSC. Real distorted videos with corresponding ground truth are recorded simultaneously via a well-designed beam-splitterbased acquisition system. BS-RSC contains various motions of both camera and objects in dynamic scenes. Further, an RS correction model with adaptive warping is proposed. Our model can warp the learned RS features into global shutter counterparts adaptively with predicted multiple displacement fields. These warped features are aggregated and then reconstructed into high-quality global shutter frames in a coarse-to-fine strategy. Experimental results demonstrate the effectiveness of the proposed method, and our dataset can improve the model's ability to remove the RS effects in the real world. The project is available at https://github.com/ljzycmd/BSRSC.
引用
收藏
页码:17764 / 17772
页数:9
相关论文
共 35 条
[1]   From two rolling shutters to one global shutter [J].
Albl, Cenek ;
Kukelova, Zuzana ;
Larsson, Viktor ;
Polic, Michal ;
Pajdla, Tomas ;
Schindler, Konrad .
2020 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 2020, :2502-2510
[2]   Rolling Shutter Camera Absolute Pose [J].
Albl, Cenek ;
Kukelova, Zuzana ;
Larsson, Viktor ;
Pajdla, Tomas .
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 2020, 42 (06) :1439-1452
[3]   Rolling shutter absolute pose problem with known vertical direction [J].
Albl, Cenek ;
Kukelova, Zuzana ;
Pajdla, Tomas .
2016 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 2016, :3355-3363
[4]  
Albl C, 2015, PROC CVPR IEEE, P2292, DOI 10.1109/CVPR.2015.7298842
[5]  
[Anonymous], 2021, CVPR, DOI DOI 10.1109/CVPR46437.2021.01212
[6]  
[Anonymous], 2020, CVPR, DOI DOI 10.1109/CVPR42600.2020.00583
[7]  
[Anonymous], 2017, CVPR
[8]  
Bahdanau D, 2016, Arxiv, DOI arXiv:1409.0473
[9]   Toward Real-World Single Image Super-Resolution: A New Benchmark and A New Model [J].
Cai, Jianrui ;
Zeng, Hui ;
Yong, Hongwei ;
Cao, Zisheng ;
Zhang, Lei .
2019 IEEE/CVF INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV 2019), 2019, :3086-3095
[10]   Rolling Shutter Camera Relative Pose: Generalized Epipolar Geometry [J].
Dai, Yuchao ;
Li, Hongdong ;
Kneip, Laurent .
2016 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 2016, :4132-4140