Glancing Text and Vision Regularized Training to Enhance Machine Translation

Bilingual parallel sentences, combined with visual annotations, created an innovative machine translation scenario within the encoder-decoder framework, known as multimodal machine translation. In generally, it was encoded as an additional visual representation to enhance the dependent-time context vector when generating the target translation word by word. However, this approach only simulated the consistency between the visual annotation and the source language and did not sufficiently consider the consistency among the source language, the target language, and the visual context. To address this problem, we proposed a novel method that adds visual features to both the encoder and decoder. In the encoder, we designed a cross-modal correlation mechanism to effectively integrate textual and visual information. In the decoder, we designed a multimodal graph to enhance the related information of vision and text. Experimental results showed that the proposed approach significantly improved translation performance compared to strong baselines for the English-German/French language pairs. The ablation study further confirmed the effectiveness of the proposed approach in improving translation quality.