Disparity-Aware Reference Frame Generation Network for Multiview Video Coding

被引:4
作者
Lei, Jianjun [1 ]
Zhang, Zongqian [1 ,2 ]
Pan, Zhaoqing [1 ]
Liu, Dong [3 ]
Liu, Xiangrui [1 ]
Chen, Ying [4 ]
Ling, Nam [5 ]
机构
[1] Tianjin Univ, Sch Elect & Informat Engn, Tianjin 300072, Peoples R China
[2] Alibaba Grp, Hangzhou 310052, Peoples R China
[3] Univ Sci & Technol China, CAS Key Lab Technol Geo Spatial Informat Proc & A, Hefei 230027, Peoples R China
[4] Alibaba Grp, Hangzhou 310052, Peoples R China
[5] Santa Clara Univ, Dept Comp Sci & Engn, Santa Clara, CA 95053 USA
基金
中国国家自然科学基金;
关键词
Image coding; Video coding; Deep learning; Image reconstruction; Estimation; Encoding; Task analysis; Multiview video coding; reference frame generation; disparity-aware alignment; DAG-Net; 3D-HEVC; VIEW SYNTHESIS; PREDICTION; EXTENSIONS;
D O I
10.1109/TIP.2022.3183436
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Multiview video coding (MVC) aims to compress the multiview video through the elimination of video redundancies, where the quality of the reference frame directly affects the compression efficiency. In this paper, we propose a deep virtual reference frame generation method based on a disparity-aware reference frame generation network (DAG-Net) to transform the disparity relationship between different viewpoints and generate a more reliable reference frame. The proposed DAG-Net consists of a multi-level receptive field module, a disparity-aware alignment module, and a fusion reconstruction module. First, a multi-level receptive field module is designed to enlarge the receptive field, and extract the multi-scale deep features of the temporal and inter-view reference frames. Then, a disparity-aware alignment module is proposed to learn the disparity relationship, and perform disparity shift on the inter-view reference frame to align it with the temporal reference frame. Finally, a fusion reconstruction module is utilized to fuse the complementary information and generate a more reliable virtual reference frame. Experiments demonstrate that the proposed reference frame generation method achieves superior performance for multiview video coding.
引用
收藏
页码:4515 / 4526
页数:12
相关论文
共 50 条
[41]   Deep Inter Prediction via Reference Frame Interpolation for Blurry Video Coding [J].
Zhu, Zezhi ;
Zhao, Lili ;
Lin, Xuhu ;
Guo, Xuezhou ;
Chen, Jianwen .
2021 INTERNATIONAL CONFERENCE ON VISUAL COMMUNICATIONS AND IMAGE PROCESSING (VCIP), 2021,
[42]   A LOW-COMPLEXITY AND LOSSLESS REFERENCE FRAME ENCODER ALGORITHM FOR VIDEO CODING [J].
Silveira, Dieison ;
Povala, Guilherme ;
Amaral, Livia ;
Zatt, Bruno ;
Agostini, Luciano ;
Porto, Marcelo .
2014 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), 2014,
[43]   Multi-Level Pipelined Parallel Hardware Architecture for High Throughput Motion and Disparity Estimation in Multiview Video Coding [J].
Zatt, Bruno ;
Shafique, Muhammad ;
Bampi, Sergio ;
Henkel, Joerg .
2011 DESIGN, AUTOMATION & TEST IN EUROPE (DATE), 2011, :1448-1453
[44]   Generative Adversarial Network-Based Frame Extrapolation for Video Coding [J].
Lin, Jianping ;
Liu, Dong ;
Li, Houqiang ;
Wu, Feng .
2018 IEEE INTERNATIONAL CONFERENCE ON VISUAL COMMUNICATIONS AND IMAGE PROCESSING (IEEE VCIP), 2018,
[45]   PROJECTIVE RECTIFICATION-BASED VIEW INTERPOLATION FOR MULTIVIEW VIDEO CODING AND FREE VIEWPOINT GENERATION [J].
Xiu, Xiaoyu ;
Liang, Jie .
PCS: 2009 PICTURE CODING SYMPOSIUM, 2009, :349-352
[46]   Reference frame list optimization algorithm in video coding by quality enhancement of the nearest picture [J].
Huo J. ;
Qiu R. ;
Ma Y. ;
Yang F. .
Tongxin Xuebao/Journal on Communications, 2022, 43 (11) :136-147
[47]   A Reference Frame Compression Scheme via AVS Perceptual Lossless Compression in Video Coding [J].
Wu, Meng ;
Chen, Huanbang ;
Yang, Fuzheng ;
Yang, Haitao ;
Feng, Junkai .
2024 INTERNATIONAL CONFERENCE ON UBIQUITOUS COMMUNICATION, UCOM 2024, 2024, :264-268
[48]   Efficient reference frame compression scheme for video coding systems: algorithm and VLSI design [J].
Silveira, Dieison ;
Povala, Guilherme ;
Amaral, Livia ;
Zatt, Bruno ;
Agostini, Luciano ;
Porto, Marcelo .
JOURNAL OF REAL-TIME IMAGE PROCESSING, 2019, 16 (02) :391-411
[49]   MOTION HINTS COMPENSATED PREDICTION AS A REFERENCE FRAME FOR HIGH EFFICIENCY VIDEO CODING (HEVC) [J].
Ahmmed, Ashek ;
Hannuksela, Miska M. ;
Gabbouj, Moncef .
2016 24TH EUROPEAN SIGNAL PROCESSING CONFERENCE (EUSIPCO), 2016, :923-927
[50]   Efficient reference frame compression scheme for video coding systems: algorithm and VLSI design [J].
Dieison Silveira ;
Guilherme Povala ;
Lívia Amaral ;
Bruno Zatt ;
Luciano Agostini ;
Marcelo Porto .
Journal of Real-Time Image Processing, 2019, 16 :391-411