A Dynamic Frame Selection Framework for Fast Video Recognition

被引:24
作者
Wu, Zuxuan [1 ]
Li, Hengduo [2 ]
Xiong, Caiming [3 ]
Jiang, Yu-Gang [1 ]
Davis, Larry Steven [2 ]
机构
[1] Fudan Univ, Sch Comp Sci, Shanghai 200433, Peoples R China
[2] Univ Maryland, Dept Comp Sci, College Pk, MD 20742 USA
[3] Salesforce Res, Palo Alto, CA 94301 USA
关键词
Computational modeling; Three-dimensional displays; Video sequences; Two dimensional displays; Computational efficiency; Standards; Electronic mail; Video classification; conditional computation; deep neural networks; reinforcement learning;
D O I
10.1109/TPAMI.2020.3029425
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
We introduce AdaFrame, a conditional computation framework that adaptively selects relevant frames on a per-input basis for fast video recognition. AdaFrame, which contains a Long Short-Term Memory augmented with a global memory to provide context information, operates as an agent to interact with video sequences aiming to search over time which frames to use. Trained with policy search methods, at each time step, AdaFrame computes a prediction, decides where to observe next, and estimates a utility, i.e., expected future rewards, of viewing more frames in the future. Exploring predicted utilities at testing time, AdaFrame is able to achieve adaptive lookahead inference so as to minimize the overall computational cost without incurring a degradation in accuracy. We conduct extensive experiments on two large-scale video benchmarks, FCVID and ActivityNet. With a vanilla ResNet-101 model, AdaFrame achieves similar performance of using all frames while only requiring, on average, 8.21 and 8.65 frames on FCVID and ActivityNet, respectively. We also demonstrate AdaFrame is compatible with modern 2D and 3D networks for video recognition. Furthermore, we show, among other things, learned frame usage can reflect the difficulty of making prediction decisions both at instance-level within the same class and at class-level among different categories.
引用
收藏
页码:1699 / 1711
页数:13
相关论文
共 50 条
[41]   Label-Guided Dynamic Spatial-Temporal Fusion for Video-Based Facial Expression Recognition [J].
Zhang, Ziyang ;
Tian, Xiang ;
Zhang, Yuan ;
Guo, Kailing ;
Xu, Xiangmin .
IEEE TRANSACTIONS ON MULTIMEDIA, 2024, 26 :10503-10513
[42]   Shots segmentation-based optimized dual-stream framework for robust human activity recognition in surveillance video [J].
Hussain, Altaf ;
Khan, Samee Ullah ;
Khan, Noman ;
Ullah, Waseem ;
Alkhayyat, Ahmed ;
Alharbi, Meshal ;
Baik, Sung Wook .
ALEXANDRIA ENGINEERING JOURNAL, 2024, 91 :632-647
[43]   A FRAMEWORK FOR FULLY CONFIGURABLE VIDEO CODING [J].
Richardson, Iain ;
Kannangara, Sampath ;
Bystrom, Maja ;
Philp, James ;
de Frutos Lopez, Manuel .
PCS: 2009 PICTURE CODING SYMPOSIUM, 2009, :145-+
[44]   Advancing Learned Video Compression With In-Loop Frame Prediction [J].
Yang, Ren ;
Timofte, Radu ;
Gool, Luc Van .
IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, 2023, 33 (05) :2410-2423
[45]   DVD: Constructing a Discriminative Video Descriptor by Convolving Frame Features [J].
Bo, Yang ;
Chen, Yixin ;
He, Wenbo ;
Xiang, Jie .
2018 IEEE FOURTH INTERNATIONAL CONFERENCE ON MULTIMEDIA BIG DATA (BIGMM), 2018,
[46]   A Motion Refinement Network With Local Compensation for Video Frame Interpolation [J].
Wang, Kaiqiao ;
Liu, Peng .
IEEE ACCESS, 2023, 11 :103092-103101
[47]   LOOKING-AHEAD: NEURAL FUTURE VIDEO FRAME PREDICTION [J].
Zhang, Changxu ;
Chen, Tong ;
Liu, Haojie ;
Shen, Qiu ;
Ma, Zhan .
2019 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 2019, :1975-1979
[48]   Residual Learning of Video Frame Interpolation Using Convolutional LSTM [J].
Suzuki, Keito ;
Ikehara, Masaaki .
IEEE ACCESS, 2020, 8 :134185-134193
[49]   A Frame-based Decision Pooling Method for Video Classification [J].
Mohanty, Ambika Ashirvad ;
Vaibhav, Bipul ;
Sethi, Amit .
2013 ANNUAL IEEE INDIA CONFERENCE (INDICON), 2013,
[50]   Deep Unsupervised Key Frame Extraction for Efficient Video Classification [J].
Tang, Hao ;
Ding, Lei ;
Wu, Songsong ;
Ren, Bin ;
Sebe, Nicu ;
Rota, Paolo .
ACM TRANSACTIONS ON MULTIMEDIA COMPUTING COMMUNICATIONS AND APPLICATIONS, 2023, 19 (03)