Distilling Privileged Knowledge for Anomalous Event Detection From Weakly Labeled Videos

被引:6
作者
Liu, Tianshan [1 ]
Lam, Kin-Man [1 ,2 ]
Kong, Jun [3 ]
机构
[1] Hong Kong Polytech Univ, Dept Elect & Informat Engn, Hong Kong, Peoples R China
[2] Ctr Adv Reliabil & Safety, Hong Kong, Peoples R China
[3] Jiangnan Univ, Key Lab Adv Proc Control Light Ind, Minist Educ, Wuxi 214122, Peoples R China
关键词
Videos; Task analysis; Feature extraction; Training; Knowledge engineering; Anomaly detection; Annotations; Privileged knowledge distillation (KD); teacher-student model; video anomaly detection (VAD); weakly supervised learning;
D O I
10.1109/TNNLS.2023.3263966
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Weakly supervised video anomaly detection (WS-VAD) aims to identify the snippets involving anomalous events in long untrimmed videos, with solely video-level binary labels. A typical paradigm among the existing WS-VAD methods is to employ multiple modalities as inputs, e.g., RGB, optical flow, and audio, as they can provide sufficient discriminative clues that are robust to the diverse, complicated real-world scenes. However, such a pipeline has high reliance on the availability of multiple modalities and is computationally expensive and storage demanding in processing long sequences, which limits its use in some applications. To address this dilemma, we propose a privileged knowledge distillation (KD) framework dedicated to the WS-VAD task, which can maintain the benefits of exploiting additional modalities, while avoiding the need for using multimodal data in the inference phase. We argue that the performance of the privileged KD framework mainly depends on two factors: 1) the effectiveness of the multimodal teacher network and 2) the completeness of the useful information transfer. To obtain a reliable teacher network, we propose a cross-modal interactive learning strategy and an anomaly normal discrimination loss, which target learning task-specific cross-modal features and encourage the separability of anomalous and normal representations, respectively. Furthermore, we design both representation-and logits-level distillation loss functions, which force the unimodal student network to distill abundant privileged knowledge from the well-trained multimodal teacher network, in a snippet-to-video fashion. Extensive experimental results on three public benchmarks demonstrate that the proposed privileged KD framework can train a lightweight yet effective detector, for localizing anomaly events under the supervision of video-level annotations.
引用
收藏
页码:12627 / 12641
页数:15
相关论文
共 65 条
  • [1] UBnormal: New Benchmark for Supervised Open-Set Video Anomaly Detection
    Acsintoae, Andra
    Florescu, Andrei
    Georgescu, Mariana-Iuliana
    Mare, Tudor
    Sumedrea, Paul
    Ionescu, Radu Tudor
    Khan, Fahad Shahbaz
    Shah, Mubarak
    [J]. 2022 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2022), 2022, : 20111 - 20121
  • [2] Barbalau A., 2022, ARXIV
  • [3] Multiple instance learning: A survey of problem characteristics and applications
    Carbonneau, Marc-Andre
    Cheplygina, Veronika
    Granger, Eric
    Gagnon, Ghyslain
    [J]. PATTERN RECOGNITION, 2018, 77 : 329 - 353
  • [4] Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
    Carreira, Joao
    Zisserman, Andrew
    [J]. 30TH IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2017), 2017, : 4724 - 4733
  • [5] Contrastive Attention for Video Anomaly Detection
    Chang, Shuning
    Li, Yanchao
    Shen, Shengmei
    Feng, Jiashi
    Zhou, Zhiying
    [J]. IEEE TRANSACTIONS ON MULTIMEDIA, 2022, 24 : 4067 - 4076
  • [6] Distilling Audio-Visual Knowledge by Compositional Contrastive Learning
    Chen, Yanbei
    Xian, Yongqin
    Koepke, A. Sophia
    Shan, Ying
    Akata, Zeynep
    [J]. 2021 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, CVPR 2021, 2021, : 7012 - 7021
  • [7] MARS: Motion-Augmented RGB Stream for Action Recognition
    Crasto, Nieves
    Weinzaepfel, Philippe
    Alahari, Karteek
    Schmid, Cordelia
    [J]. 2019 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2019), 2019, : 7874 - 7883
  • [8] Learning an Augmented RGB Representation with Cross-Modal Knowledge Distillation for Action Detection
    Dai, Rui
    Das, Srijan
    Bremond, Francois
    [J]. 2021 IEEE/CVF INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV 2021), 2021, : 13033 - 13044
  • [9] Learning Spatiotemporal Features with 3D Convolutional Networks
    Du Tran
    Bourdev, Lubomir
    Fergus, Rob
    Torresani, Lorenzo
    Paluri, Manohar
    [J]. 2015 IEEE INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV), 2015, : 4489 - 4497
  • [10] Anomaly Detection With Bidirectional Consistency in Videos
    Fang, Zhiwen
    Liang, Jiafei
    Zhou, Joey Tianyi
    Xiao, Yang
    Yang, Feng
    [J]. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, 2022, 33 (03) : 1079 - 1092