Video reconstruction from a single motion blurred image using learned dynamic phase coding

被引:4
作者
Yosef, Erez [1 ]
Elmalem, Shay [1 ,2 ]
Giryes, Raja [1 ]
机构
[1] Tel Aviv Univ, Tel Aviv, Israel
[2] Weizmann Inst Sci, Rehovot, Israel
基金
欧洲研究理事会;
关键词
CODED-APERTURE; EXPOSURE; DEPTH;
D O I
10.1038/s41598-023-40297-0
中图分类号
O [数理科学和化学]; P [天文学、地球科学]; Q [生物科学]; N [自然科学总论];
学科分类号
07 ; 0710 ; 09 ;
摘要
Video reconstruction from a single motion-blurred image is a challenging problem, which can enhance the capabilities of existing cameras. Recently, several works addressed this task using conventional imaging and deep learning. Yet, such purely digital methods are inherently limited, due to direction ambiguity and noise sensitivity. Some works attempt to address these limitations with non-conventional image sensors, however, such sensors are extremely rare and expensive. To circumvent these limitations by simpler means, we propose a hybrid optical-digital method for video reconstruction that requires only simple modifications to existing optical systems. We use learned dynamic phase-coding in the lens aperture during image acquisition to encode motion trajectories, which serve as prior information for the video reconstruction process. The proposed computational camera generates a sharp frame burst of the scene at various frame rates from a single coded motion-blurred image, using an image-to-video convolutional neural network. We present advantages and improved performance compared to existing methods, with both simulations and a real-world camera prototype. We extend our optical coding to video frame interpolation and present robust and improved results for noisy videos.
引用
收藏
页数:12
相关论文
共 42 条
[1]  
[Anonymous], iDS UI-3590CP. iDS UI-3590CP CMOS Sensor
[2]   Video from Stills: Lensless Imaging with Rolling Shutter [J].
Antipa, Nick ;
Oare, Patrick ;
Bostan, Emrah ;
Ng, Ren ;
Waller, Laura .
2019 IEEE INTERNATIONAL CONFERENCE ON COMPUTATIONAL PHOTOGRAPHY (ICCP), 2019,
[3]   Video reconstruction by spatio-temporal fusion of blurred-coded image pair [J].
Anupama, S. ;
Shedligeri, Prasan ;
Pal, Abhishek ;
Mitra, Kaushik .
2020 25TH INTERNATIONAL CONFERENCE ON PATTERN RECOGNITION (ICPR), 2021, :7953-7960
[4]  
Argaw DM, 2021, AAAI CONF ARTIF INTE, V35, P901
[5]  
Bao W. etal, 2019, IEEE C COMPUTER VISI
[6]   Motion deblurring using spatiotemporal phase aperture coding [J].
Elmalem, Shay ;
Giryes, Raja ;
Marom, Emanuel .
OPTICA, 2020, 7 (10) :1332-1340
[7]   Learned phase coded aperture for the benefit of depth of field extension [J].
Elmalem, Shay ;
Giryes, Raja ;
Marom, Emanuel .
OPTICS EXPRESS, 2018, 26 (12) :15316-15331
[8]   Depth Estimation From a Single Image Using Deep Learned Phase Coded Mask [J].
Haim, Harel ;
Elmalem, Shay ;
Giryes, Raja ;
Bronstein, Alex M. ;
Marom, Emanuel .
IEEE TRANSACTIONS ON COMPUTATIONAL IMAGING, 2018, 4 (03) :298-310
[9]  
Holloway J., 2012, IEEE Int. Conf. Comput. Photogr, DOI [10.1109/ICCPhot.2012.6215211, DOI 10.1109/ICCPHOT.2012.6215211]
[10]   Densely Connected Convolutional Networks [J].
Huang, Gao ;
Liu, Zhuang ;
van der Maaten, Laurens ;
Weinberger, Kilian Q. .
30TH IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2017), 2017, :2261-2269