Modality correlation-based video summarization

被引:7
|
作者
Wang, Xingrun [1 ]
Nie, Xiushan [2 ]
Liu, Xingbo [1 ]
Wang, Binze [3 ]
Yin, Yilong [4 ]
机构
[1] Shandong Univ, Sch Comp Sci & Technol, Jinan 250101, Shandong, Peoples R China
[2] Shandong Jianzhu Univ, Sch Comp Sci & Technol, Jinan 250101, Shandong, Peoples R China
[3] Changan Univ, Coll Geol Engn & Geomat, Xian 710054, Peoples R China
[4] Shandong Univ, Sch Software Engn, Jinan 250101, Shandong, Peoples R China
基金
中国国家自然科学基金;
关键词
Video summarization; Modality correlation; Modality-specific information; Attention mechanism;
D O I
10.1007/s11042-020-08690-3
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
Video summarization is an important technique to help us browse, store, and retrieve a rapidly increasing amount of video data, which extracts frames or shots from the original video. Text information covers important content of a video, and thus a summarization can be generated by exploring the correlation between the frame and text. In this study, we propose a video summarization method based on the modality correlation. With this method, we first learn the correlation between the text and frame in the respective space, and then fuse two correlations to obtain the importance score of each shot. Finally, video shots that have a high importance score are chosen as the video summarization. Compared to previous methods that seldom apply text to generate the video summarization, or only use the latent common information between text and frame, the proposed method fully utilizes not only the latent common but also modality-specific information for a video summarization. Experiments were conducted on the TVSum50 dataset, and the results verify the effectiveness of our proposed approach.
引用
收藏
页码:33875 / 33890
页数:16
相关论文
共 50 条
  • [41] A mode correlation-based fractional pixel motion estimation for H.264 video coding
    Fang Jian
    Zheng Wei
    Zhang Ding
    Wang Kuang
    ASICON 2007: 2007 7TH INTERNATIONAL CONFERENCE ON ASIC, VOLS 1 AND 2, PROCEEDINGS, 2007, : 938 - 941
  • [42] Spatial Correlation-Based Motion-Vector Prediction for Video-Coding Efficiency Improvement
    Jiang, Xiantao
    Song, Tian
    Katayama, Takafumi
    Leu, Jenq-Shiou
    SYMMETRY-BASEL, 2019, 11 (02):
  • [43] Correlation-Based Feature Selection and Regression
    Cui, Yue
    Lin, Jesse S.
    Zhang, Shiliang
    Luo, Suhuai
    Tian, Qi
    ADVANCES IN MULTIMEDIA INFORMATION PROCESSING-PCM 2010, PT I, 2010, 6297 : 25 - +
  • [44] Occlusion handling in correlation-based matching
    Chambon, Sylvie
    Crouzil, Alain
    TRAITEMENT DU SIGNAL, 2007, 24 (06) : 429 - 446
  • [45] Ensemble Learning with Correlation-Based Penalty
    Liu, Yong
    Zhao, Qiangfu
    Pei, Yan
    2014 IEEE 12TH INTERNATIONAL CONFERENCE ON DEPENDABLE, AUTONOMIC AND SECURE COMPUTING (DASC)/2014 IEEE 12TH INTERNATIONAL CONFERENCE ON EMBEDDED COMPUTING (EMBEDDEDCOM)/2014 IEEE 12TH INTERNATIONAL CONF ON PERVASIVE INTELLIGENCE AND COMPUTING (PICOM), 2014, : 350 - 353
  • [46] Correlation-based block truncation coding
    Cai, ZQ
    ELECTRONICS LETTERS, 1996, 32 (25) : 2305 - 2306
  • [47] Correlation-based incremental visual tracking
    Kim, Minyoung
    PATTERN RECOGNITION, 2012, 45 (03) : 1050 - 1060
  • [48] CPR: Correlation-based Page Remapping
    Namkoong, Hojung
    Kim, Jungrae
    2023 20TH INTERNATIONAL SOC DESIGN CONFERENCE, ISOCC, 2023, : 309 - 310
  • [49] Scale correlation-based edge detection
    Bao, P
    Lei, Z
    PROCEEDINGS VIPROMCOM-2002, 2002, : 345 - 350
  • [50] Quantile Correlation-based Variable Selection
    Tang, Wenlu
    Xie, Jinhan
    Lin, Yuanyuan
    Tang, Niansheng
    JOURNAL OF BUSINESS & ECONOMIC STATISTICS, 2022, 40 (03) : 1081 - 1093