Low-Resource Neural Machine Translation Using XLNet Pre-training Model

The methods to improve the quality of low-resource neural machine translation (NMT) include: change the token granularity to reduce the number of low-frequency words; generate pseudo-parallel corpus from large-scale monolingual data to optimize model parameters; Use the auxiliary knowledge of pre-trained model to train NMT model. However, reducing token granularity will result in a large number of invalid operations and increase the complexity of local reordering on the target side. Pseudo-parallel corpus contains noise affect model convergence. Pre-training methods also limit translation quality due to the human error and the assumption of conditional independence. Therefore, we proposed a XLNet based pre-training method, that corrects the defects of the pre-training model, and enhance NMT model for context feature extraction. Experiments are carried out on CCMT2019 Mongolian-Chinese (Mo-Zh), Uyghur-Chinese (Ug-Zh) and Tibetan-Chinese (Ti-Zh) tasks, the results show that the generalization ability and BLEU scores of our method are improved compared with the baseline, which fully verifies the effectiveness of the method.