A Dynamic Frame Selection Framework for Fast Video Recognition

被引:24
作者
Wu, Zuxuan [1 ]
Li, Hengduo [2 ]
Xiong, Caiming [3 ]
Jiang, Yu-Gang [1 ]
Davis, Larry Steven [2 ]
机构
[1] Fudan Univ, Sch Comp Sci, Shanghai 200433, Peoples R China
[2] Univ Maryland, Dept Comp Sci, College Pk, MD 20742 USA
[3] Salesforce Res, Palo Alto, CA 94301 USA
关键词
Computational modeling; Three-dimensional displays; Video sequences; Two dimensional displays; Computational efficiency; Standards; Electronic mail; Video classification; conditional computation; deep neural networks; reinforcement learning;
D O I
10.1109/TPAMI.2020.3029425
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
We introduce AdaFrame, a conditional computation framework that adaptively selects relevant frames on a per-input basis for fast video recognition. AdaFrame, which contains a Long Short-Term Memory augmented with a global memory to provide context information, operates as an agent to interact with video sequences aiming to search over time which frames to use. Trained with policy search methods, at each time step, AdaFrame computes a prediction, decides where to observe next, and estimates a utility, i.e., expected future rewards, of viewing more frames in the future. Exploring predicted utilities at testing time, AdaFrame is able to achieve adaptive lookahead inference so as to minimize the overall computational cost without incurring a degradation in accuracy. We conduct extensive experiments on two large-scale video benchmarks, FCVID and ActivityNet. With a vanilla ResNet-101 model, AdaFrame achieves similar performance of using all frames while only requiring, on average, 8.21 and 8.65 frames on FCVID and ActivityNet, respectively. We also demonstrate AdaFrame is compatible with modern 2D and 3D networks for video recognition. Furthermore, we show, among other things, learned frame usage can reflect the difficulty of making prediction decisions both at instance-level within the same class and at class-level among different categories.
引用
收藏
页码:1699 / 1711
页数:13
相关论文
共 50 条
  • [31] FFFN: Frame-By-Frame Feedback Fusion Network for Video Super-Resolution
    Zhu, Jian
    Zhang, Qingwu
    Fei, Lunke
    Cai, Ruichu
    Xie, Yuan
    Sheng, Bin
    Yang, Xiaokang
    [J]. IEEE TRANSACTIONS ON MULTIMEDIA, 2023, 25 : 6821 - 6835
  • [32] Short-Term Action Learning for Video Action Recognition
    Ting-Long, Liu
    [J]. IEEE ACCESS, 2024, 12 : 30867 - 30875
  • [33] Similarity-Aware CNN for Efficient Video Recognition at the Edge
    Sabet, Amin
    Hare, Jonathon
    Al-Hashimi, Bashir M.
    Merrett, Geoff, V
    [J]. IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS, 2022, 41 (11) : 4901 - 4914
  • [34] Active Learning for Video Classification with Frame Level Queries
    Goswami, Debanjan
    Chakraborty, Shayok
    [J]. 2023 INTERNATIONAL JOINT CONFERENCE ON NEURAL NETWORKS, IJCNN, 2023,
  • [35] Nonlinear analysis and synthesis of video images using deep dynamic bottleneck neural networks for face recognition
    Moghadam, Saeed Montazeri
    Seyyedsalehi, Seyyed Ali
    [J]. NEURAL NETWORKS, 2018, 105 : 304 - 315
  • [36] EMO-MoviNet: Enhancing Action Recognition in Videos with EvoNorm, Mish Activation, and Optimal Frame Selection for Efficient Mobile Deployment
    Hussain, Tarique
    Memon, Zulfiqar Ali
    Qureshi, Rizwan
    Alam, Tanvir
    [J]. SENSORS, 2023, 23 (19)
  • [37] Rethinking Lightweight: Multiple Angle Strategy for Efficient Video Action Recognition
    Chen, Jianyu
    Wang, Zhongyuan
    Zeng, Kangli
    He, Zheng
    Xiong, Zixiang
    [J]. IEEE SIGNAL PROCESSING LETTERS, 2022, 29 : 498 - 502
  • [38] Metric-Based Attention Feature Learning for Video Action Recognition
    Kim, Dae Ha
    Anvarov, Fazliddin
    Lee, Jun Min
    Song, Byung Cheol
    [J]. IEEE ACCESS, 2021, 9 : 39218 - 39228
  • [39] Dynamic Sampling Networks for Efficient Action Recognition in Videos
    Zheng, Yin-Dong
    Liu, Zhaoyang
    Lu, Tong
    Wang, Limin
    [J]. IEEE TRANSACTIONS ON IMAGE PROCESSING, 2020, 29 (29) : 7970 - 7983
  • [40] Deep Spiking Neural Network for Video-Based Disguise Face Recognition Based on Dynamic Facial Movements
    Liu, Daqi
    Bellotto, Nicola
    Yue, Shigang
    [J]. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, 2020, 31 (06) : 1843 - 1855