A Viewport Prediction Framework for Panoramic Videos

被引:6
|
作者
Tang, Jinting [1 ,2 ]
Huo, Yongkai [1 ,2 ]
Yang, Shaoshi [3 ,4 ]
Jiang, Jianmin [1 ,2 ]
机构
[1] Shenzhen Univ, Sch Comp Sci & Software Engn, Natl Engn Lab Big Data Syst Comp Technol, Shenzhen 518060, Peoples R China
[2] Shenzhen Univ, Sch Comp Sci & Software Engn, Res Inst Future Media Comp, Shenzhen 518060, Peoples R China
[3] Beijing Univ Posts & Telecommun, Sch Informat & Commun Engn, Beijing 100876, Peoples R China
[4] Beijing Univ Posts & Telecommun, Minist Educ, Key Lab Universal Wireless Commun, Beijing 100876, Peoples R China
来源
2020 INTERNATIONAL JOINT CONFERENCE ON NEURAL NETWORKS (IJCNN) | 2020年
基金
中国国家自然科学基金;
关键词
panoramic video; viewport prediction; object tracking; deep learning;
D O I
10.1109/ijcnn48605.2020.9207562
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Panoramic video is considered to be an attractive video format, since it provides the viewers with an immersive experience, such as virtual reality (VR) gaming. However, the viewers only focus on part of panoramic video, which is referred to as viewport. Hence, the resources consumed for distributing the remaining part of the panoramic video are wasted. It is intuitive to only deliver the video data within this viewport for reducing the distribution cost. Empirically, viewports within a time interval are highly correlated, hence the historical trajectory may be used for predicting the future viewports. On the other hand, a viewer tends to sustain attention on a specific object in a panoramic video. Motivated by these findings, we propose a deep learning-based viewport Prediction scheme, namely HOP, where the Historical viewport trajectory of viewers and Object tracking are jointly exploited by the long short-term memory (LSTM) networks. Additionally, our solution is capable of predicting multiple future viewports, while a single viewport prediction was supported by the state-of-the-art contributions. Simulation results show that our proposed HOP scheme outperforms the benchmarkers by up to 33.5% in terms of the prediction error.
引用
收藏
页数:8
相关论文
共 50 条
  • [1] Viewport Prediction for 360° Videos: A Clustering Approach
    Nasrabadi, Afshin Taghavi
    Samiei, Aliehsan
    Prakash, Ravi
    NOSSDAV '20: PROCEEDINGS OF THE 2020 WORKSHOP ON NETWORK AND OPERATING SYSTEM SUPPORT FOR DIGITAL AUDIO AND VIDEO, 2020, : 34 - 39
  • [2] Viewport Prediction for Panoramic Video with Multi-CNN
    Li, Xiao
    Wang, Siyi
    Zhu, Chen
    Song, Li
    Xie, Rong
    Zhang, Wenjun
    2019 IEEE INTERNATIONAL SYMPOSIUM ON BROADBAND MULTIMEDIA SYSTEMS AND BROADCASTING (BMSB), 2019,
  • [3] Panoramic Video Inter Frame Prediction and Viewport Prediction Based on Background Modeling
    Wang, Changli
    Wang, Xingtao
    Wu, Kaixin
    Fan, Xiaopeng
    ADVANCED INTELLIGENT COMPUTING TECHNOLOGY AND APPLICATIONS, PT X, ICIC 2024, 2024, 14871 : 261 - 272
  • [4] Multi-stream-Based Low-Latency Viewport Switching Scheme for Panoramic Videos
    Wang, Yong
    Man, Hengyu
    Wang, Xingtao
    Fan, Xiaopeng
    PATTERN RECOGNITION AND COMPUTER VISION, PRCV 2023, PT XI, 2024, 14435 : 104 - 116
  • [5] VIEWPORT-ORIENTED PANORAMIC IMAGE INPAINTING
    Shang, Zhuoyi
    Liu, Yanwei
    Li, Guoyi
    Zhang, Yunjian
    Miao, Jingbo
    Liu, Jinxia
    Wang, Liming
    2022 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING, ICIP, 2022, : 3031 - 3035
  • [6] Tiled streaming for layered 3D virtual reality videos with viewport prediction
    Hong-Yun Chen
    Chow-Sing Lin
    Multimedia Tools and Applications, 2022, 81 : 13867 - 13888
  • [7] Multi-source Information Perception and Prediction for Panoramic Videos
    Qu, Chenxin
    Li, Kexin
    Che, Xiaoping
    Chang, Enyao
    Zhang, Zhongwei
    ADVANCES IN COMPUTER GRAPHICS, CGI 2023, PT I, 2024, 14495 : 451 - 462
  • [8] Tiled streaming for layered 3D virtual reality videos with viewport prediction
    Chen, Hong-Yun
    Lin, Chow-Sing
    MULTIMEDIA TOOLS AND APPLICATIONS, 2022, 81 (10) : 13867 - 13888
  • [9] Trajectory-Based Viewport Prediction for 360-Degree Virtual Reality Videos
    Petrangeli, Stefano
    Simon, Gwendal
    Swaminathan, Viswanathan
    2018 IEEE INTERNATIONAL CONFERENCE ON ARTIFICIAL INTELLIGENCE AND VIRTUAL REALITY (AIVR), 2018, : 157 - 160
  • [10] PHD: A Deep Learning Based Human Detection Framework for Panoramic Videos
    Tang, Jinting
    Chen, Zhenhui
    Huo, Yongkai
    Zhang, Peichang
    2019 11TH INTERNATIONAL CONFERENCE ON WIRELESS COMMUNICATIONS AND SIGNAL PROCESSING (WCSP), 2019,