Self-Similarity Action Proposal

Temporal action proposal generation, which aims to locate temporal segments that may contain actions, is a key prepositive step of various video analysis tasks, like temporal action detection. In this letter, we present Self-Similarity Action Proposal (SSAP), a simple method that generates action proposals using the self-similarity of videos. Specifically, a basic low-level index, structural similarity, is adopted to measure the similarity between adjacent frames. Potential action boundaries are located by thresholding the similarity values and candidate action segments are successively generated by grouping the boundaries. A segment evaluation module (SEM) is further employed to score and refine the segments. The framework achieves state-of-the-art performance on THUMOS14 and competitive results on ActivityNet v1.3. Notably, on THUMOS14, it achieves over 4% improvement on the average recall at 50 proposals and 3.3% gain in mAP@0.7 when combined with an existing action classifier for temporal action detection.