Temporal graph convolutional network for multi-agent reinforcement learning of action detection

Most action detection techniques process untrimmed action videos using temporal context, without explicitly consider the inherent spatio-temporal context information, leading to limited accuracy in the cases with big spatial complexity and temporal redundancy. To this end, by intuitively dealing with the problem following a two-stage "clip selection + clip classification" scheme, this paper proposes to formulate action detection as a Markov process and builds up a multi-agent reinforcement learning framework capturing global structural relationships of videos to optimize the selection and classification, simultaneously and progressively. In particular, a temporal graph convolutional network is constructed to represent the spatio-temporal correlations of video clips, which are initialized by evenly sampling, and further adjusted via learning the rewards adaptively for multi-agent cooperation. Multi-head dot-product attention mechanism is adopted to integrate the relations of latent CNN features of interacting agents. Our framework is jointly learnt by fusing the objectives of clip selection policy and clip recognition. The proposed method comprises a novel graph convolutional network based spatio-temporal semantic observation module which captures topological features among nearby agents, and a new policy module that segments actions according to the rewards from the objectives of action recognition. Extensive experiments are conducted on ActivityNet v1.3 and THUMOS14, with 30.13% and 55.4% mAP obtained, demonstrate the applicability and superiority of our approach.