Time-frequency recurrent transformer with diversity constraint for dense video captioning

Describing a long video using multiple sentences, i.e., dense video captioning, is a very challenging task. Existing methods neglect the important fact that the actions of several tempos (a.k.a., frequencies) evolve with the time in video, and do not well handle the phrase repetition issue. Therefore, we propose a Time -Frequency recurrent Transformer with Diversity constraint (TFTD) for dense video captioning. Its basic idea is to develop a time-frequency memory module, which not only stores the history of the past sentences and corresponding video segments to consider the temporal relations, but also models the motion dependency of action patterns with different frequencies. This contributes to producing more coherent sentences that well describe the video content. Moreover, we adopt the Determinantal Point Processes (DDP) for designing the diversity loss imposed to the objective function as constraint, such that the generated sentences are diverse with less redundancy. Extensive experiments on two benchmark datasets have verified the superior performance of our approach, e.g., it achieves 11.36, 16.56, 26.16, and 3.77, in terms of BLEU@4, METEOR, CIDEr-D, and R@4, respectively, on ActivityNet Captions (20,000 videos). Besides, TFTD outperforms the most competitive alternative by 8.0% on ActivityNet Captions and 9.8% on YouCookII (2,000 videos) in terms of coherence for human evaluation.