Auxiliary criterion conversion via spatiotemporal semantic encoding and feature entropy for action recognition

Video-based action recognition in realistic scenes is a core technology for human–computer interaction and smart surveillance. Although the trajectory features with the bag of visual words have confirmed promising performance, spatiotemporal interactive information cannot be effectively encoded which is valuable for classification. To address this issue, we propose a spatiotemporal semantic feature (ST-SF) and implement the conversion of it to the auxiliary criterion based on the information entropy theory. First, we present a text-based relevance analysis method to estimate the textual labels of objects most relevant to actions, which are employed to train the more targeted detectors based on the deep network. False detections are optimized by the inter-frame cooperativity and dynamic programming to construct the valid tubes. Then, we design the ST-SF to encode the interactive information, and the concept and calculation of feature entropy are defined based on the spatial distribution of ST-SFs on the training set. Finally, we achieve a two-stage classification strategy using the resulting decision gains. Experimental results on three publicly available datasets demonstrate that our method is robust and improves upon the state-of-the-art algorithms.