Multilingual Pre-training Model-Assisted Contrastive Learning Neural Machine Translation

Since pre-training and fine-tuning have been a successful paradigm in Natural Language Processing (NLP), this paper adopts the SOTA pre-training model-CeMAT as a strong assistant for low-resource ethnic language translation tasks. Aiming at the exposure bias problem in the fine-tuning process, we use the contrastive learning framework and propose a new contrastive examples generation method, which uses self-generated predictions as contrastive examples to expose the model to errors during inference. Moreover, in order to effectively utilize the limited bilingual data in low-resource tasks, this paper proposes a dynamic training strategy to fine-tune the model, and refines the model step by step by taking word embedding norm and uncertainty as the criteria of evaluate data and model respectively. Experimental results demonstrate that our method significantly improves the quality compared to the baselines, which fully verifies the effectiveness.