ACMC: Adaptive cross-modal multi-grained contrastive learning for continuous sign language recognition

Continuous sign language recognition helps the hearing-impaired community participate in social communication by recognizing the semantics of sign language video. However, the existing CSLR methods usually only implement cross-modal alignment at the sentence level or frame level, and do not fully consider the potential impact of redundant frames and semantically independent gloss identifiers on the recognition results. In order to improve the limitations of the above methods, we propose an adaptive cross-modal multi-grained contrastive learning (ACMC) for continuous sign language recognition, which achieve more accurate cross-modal semantic alignment through a multi-grained contrast mechanism. First, the ACMC uses the frame extractor and the temporal modeling module to obtain the fine-grained and coarse-grained features of the visual modality in turn, and extracts the fine-grained and coarse-grained features of the text modality through the CLIP text encoder. Then, the ACMC adopts coarse-grained contrast and fine-grained contrast methods to effectively align the features of visual and text modalities from global and local perspectives, and alleviate the semantic interference caused by redundant frames and semantically independent gloss identifiers through cross-grained contrast. In addition, in the video frame extraction stage, we design an adaptive learning module to strengthen the features of key regions of video frames through the calculated discrete spatial feature decision matrix, and adaptively fuse the convolution features of key frames with the trajectory information between adjacent frames, thereby reducing the computational cost. Experimental results show that the proposed ACMC model achieves very competitive recognition results on sign language datasets such as PHOENIX14, PHOENIX14-T and CSL-Daily.