Video Captioning Based on C3D and Visual Elements

With the development of deep learning, the approach that extracts video feature using convolutional neural networks(CNNs) and generates sentences using recurrent neural networks(RNNs) is widely used in video captioning task. However, this direct translation ignores many intrinsic information of video, such as temporal information, motion information, and abundant visual element information. Therefore, a multi-modality video caption model based on self-adaptive frame cycle filling algorithm(AFCF-MVC) is proposed in this paper, which uses the self-adaptive feature extraction algorithm to extract the video C3D features that contain rich spatial-temporal information, and then utilizes them as inputs to the neural network. At the same time, the self-adaptive feature extraction algorithm can exploit the whole video information. Owing to the lengths of video references are different, a self-adaptive frame cycle filling algorithm is proposed to adaptively control the number of input features according to the length of the annotation sentence, which provides as many feature inputs as possible for the neural network under the premise of ensuring the complete input of the sentence. Moreover, it also plays the role of repeated learning. In order to make use of the rich visual elements of videos, the visual elements of video frames are detected by a visual detector, and then they are encoded into the network as supplementary information. Experimental results on M-VAD and MPII-MD datasets show that the proposed method not only describes the video content correctly, but also can simulate the richness of human language. © 2018, Editorial Department, Journal of South China University of Technology. All right reserved.