Semantic Video Moment Retrieval by Temporal Feature Perturbation and Refinement

Video moment retrieval (VMR) aims to locate temporal activities in untrimmed videos by sentence queries, facing a temporal bias problem. VMR models tend to over-rely on statistical regularities, instead of cross-modal semantics, and perform poorly against different distributions. Existing attempts clip/reorder video segments or overlook some samples, leading to sample integrity break and information waste, which are inappropriate when utilizing limited VMR datasets from labor-intensive labeling. In this work, without sacrificing samples' inherent value to balance performances, we develop a novel Temporal feature Perturbation and Refinement (TPR) method to augment each sample. Specifically, we perturb frame features by manipulating their time-level statistics, to diversify temporal distributions and promote more generalizable cross-modal learning. Considering the plausible moment boundary shifts brought by perturbation, we further refine final predictions by augmenting timepoint labels to candidate endpoint sets with designed query triplets. Experiments show TPR's superiority on various temporal distributions.