A Study on the Use of Attention for Explaining Video Summarization

被引:0
作者
Apostolidis, Evlampios [1 ]
Mezaris, Vasileios [1 ]
Patras, Ioannis [2 ]
机构
[1] CERTH ITI, Thessaloniki 57001, Greece
[2] Queen Mary Univ London, London E1 4NS, England
来源
PROCEEDINGS OF THE 2ND WORKSHOP ON USER-CENTRIC NARRATIVE SUMMARIZATION OF LONG VIDEOS, NARSUM 2023 | 2023年
基金
欧盟地平线“2020”;
关键词
Video summarization; Explainable AI; Attention mechanism; Explanation signals; Replacement functions; Sanity Violation;
D O I
10.1145/3607540.3617138
中图分类号
TP301 [理论、方法];
学科分类号
081202 ;
摘要
In this paper we present our study on the use of attention for explaining video summarization. We build on a recent work that formulates the task, called XAI-SUM, and we extend it by: a) taking into account two additional network architectures and b) introducing two novel explanation signals that relate to the entropy and diversity of attention weights. In total, we examine the effectiveness of seven types of explanation, using three state-of-the-art attention-based network architectures (CA-SUM, VASNet, SUM-GDA) and two datasets (SumMe, TVSum) for video summarization. The conducted evaluations show that the inherent attention weights are more suitable for explaining network architectures which integrate mechanisms for estimating attentive diversity (SUM-GDA) and uniqueness (CA-SUM). The explanation of simpler architectures (VASNet) can benefit from taking into account estimates about the strength of the input vectors, while another option is to consider the entropy of attention weights.
引用
收藏
页码:41 / 49
页数:9
相关论文
共 40 条
[1]  
Aakur Sathyanarayanan N., 2018, WORKSH 32 AAAI C ART
[2]   Explaining video summarization based on the focus of attention [J].
Apostolidis, Evlampios ;
Balaouras, Georgios ;
Mezaris, Vasileios ;
Patras, Ioannis .
2022 IEEE INTERNATIONAL SYMPOSIUM ON MULTIMEDIA (ISM), 2022, :146-150
[3]   Summarizing Videos using Concentrated Attention and Considering the Uniqueness and Diversity of the Video Frames [J].
Apostolidis, Evlampios ;
Balaouras, Georgios ;
Mezaris, Vasileios ;
Patras, Ioannis .
PROCEEDINGS OF THE 2022 INTERNATIONAL CONFERENCE ON MULTIMEDIA RETRIEVAL, ICMR 2022, 2022, :407-415
[4]   Combining Global and Local Attention with Positional Encoding for Video Summarization [J].
Apostolidis, Evlampios ;
Balaouras, Georgios ;
Mezaris, Vasileios ;
Patras, Ioannis .
23RD IEEE INTERNATIONAL SYMPOSIUM ON MULTIMEDIA (ISM 2021), 2021, :226-234
[5]   Video Summarization Using Deep Neural Networks: A Survey [J].
Apostolidis, Evlampios ;
Adamantidou, Eleni ;
Metsai, Alexandros, I ;
Mezaris, Vasileios ;
Patras, Ioannis .
PROCEEDINGS OF THE IEEE, 2021, 109 (11) :1838-1863
[6]   Excitation Backprop for RNNs [J].
Bargal, Sarah Adel ;
Zunino, Andrea ;
Kim, Donghyun ;
Zhang, Jianming ;
Murino, Vittorio ;
Sclaroff, Stan .
2018 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 2018, :1440-1449
[7]  
Chrysostomou G, 2022, PROCEEDINGS OF THE 60TH ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS (ACL 2022), VOL 1: (LONG PAPERS), P6920
[8]  
Chrysostomou G, 2021, 59TH ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS AND THE 11TH INTERNATIONAL JOINT CONFERENCE ON NATURAL LANGUAGE PROCESSING, VOL 1 (ACL-IJCNLP 2021), P477
[9]   AWeb Service for Video Summarization [J].
Collyda, Chrysa ;
Apostolidis, Konstantinos ;
Apostolidis, Evlampios ;
Adamantidou, Eleni ;
Metsai, Alexandros I. ;
Mezaris, Vasileios .
PROCEEDINGS OF THE 2020 ACM INTERNATIONAL CONFERENCE ON INTERACTIVE MEDIA EXPERIENCES, IMX 2020, 2020, :148-153
[10]  
Deng J, 2009, PROC CVPR IEEE, P248, DOI 10.1109/CVPRW.2009.5206848