Modeling Temporal Dynamics and Spatial Configurations of Actions Using Two-Stream Recurrent Neural Networks

被引:345
作者
Wang, Hongsong [1 ,3 ]
Wang, Liang [1 ,2 ,3 ]
机构
[1] Chinese Acad Sci CASIA, NLPR, CRIPAC, Beijing, Peoples R China
[2] Chinese Acad Sci, Inst Automat, CEBSIT, Beijing, Peoples R China
[3] UCAS, Beijing, Peoples R China
来源
30TH IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2017) | 2017年
基金
北京市自然科学基金; 中国国家自然科学基金;
关键词
D O I
10.1109/CVPR.2017.387
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Recently, skeleton based action recognition gains more popularity due to cost-effective depth sensors coupled with real-time skeleton estimation algorithms. Traditional approaches based on handcrafted features are limited to represent the complexity of motion patterns. Recent methods that use Recurrent Neural Networks (RNN) to handle raw skeletons only focus on the contextual dependency in the temporal domain and neglect the spatial configurations of articulated skeletons. In this paper, we propose a novel two-stream RNN architecture to model both temporal dynamics and spatial configurations for skeleton based action recognition. We explore two different structures for the temporal stream: stacked RNN and hierarchical RNN. Hierarchical RNN is designed according to human body kinematics. We also propose two effective methods to model the spatial structure by converting the spatial graph into a sequence of joints. To improve generalization of our model, we further exploit 3D transformation based data augmentation techniques including rotation and scaling transformation to transform the 3D coordinates of skeletons during training. Experiments on 3D action recognition benchmark datasets show that our method brings a considerable improvement for a variety of actions, i.e., generic actions, interaction activities and gestures.
引用
收藏
页码:3633 / 3642
页数:10
相关论文
共 55 条
  • [1] Aggarwal J. K., 2014, PRL
  • [2] [Anonymous], 2009, TPAMI
  • [3] [Anonymous], COMMUNICATIONS ACM
  • [4] [Anonymous], CVPR
  • [5] [Anonymous], 2012, 2012 IEEE COMPUTER S
  • [6] [Anonymous], PLAN ACTIVITY INTENT
  • [7] [Anonymous], 2015, ACPR
  • [8] [Anonymous], 2015, CVPR
  • [9] [Anonymous], 2014, CVPR
  • [10] [Anonymous], 2015, CVPR