Free-FreeSLT: A Gloss-Free Parameter-Free model for Sign Language Translation

Sign language translation (SLT) is a demanding task involving integrating visual and linguistic information, requiring cross-modal learning to translate visual motions into text. Current gloss-based methods employ gloss annotations for translation. Due to the availability of annotated sign language video data, gloss-based methods rely on labor-intensive and high-quality annotation work for sign language videos. To tackle this issue, we introduce a novel two-stage gloss-free sign language translation model with a parameter-free visual-language pre-training method, enhancing visual and semantic representations without introducing extra parameters. The proposed two-stage model involves: (i) During the pre-training stage, integrating Contrastive Language-Image Pre-training (CLIP) is adopted to align visual and textual features, which are then aggregated using a mean pooling mechanism; (ii) For the fine-tuning stage, parameters from the pre-trained model are inherited to enhance sign language translation. Our proposed model surpasses the leading gloss-free SLT model on PHOENIX-2014T across various n-gram levels in the BLEU score.