Multilingual Neural Machine Translation for Indic to Indic Languages

被引:5
作者
Das, Sudhansu Bala [1 ]
Panda, Divyajyoti [1 ]
Mishra, Tapas Kumar [1 ]
Patra, Bidyut Kr. [2 ]
Ekbal, Asif [3 ]
机构
[1] Natl Inst Technol NIT, Rourkela, Odisha, India
[2] Indian Inst Technol BHU, Varanasi, Uttar Pradesh, India
[3] Indian Inst Technol IIT, Patna, Bihar, India
关键词
Multilingual neural machine translation; Indic languages; language relatedness; Pivot; transliteration;
D O I
10.1145/3652026
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
The method of translation from one language to another without human intervention is known as Machine Translation (MT). Multilingual neural machine translation (MNMT) is a technique for MT that builds a single model for multiple languages. It is preferred over other approaches, since it decreases training time and improves translation in low-resource contexts, i.e., for languages that have insufficient corpus. However, goodquality MT models are yet to be built for many scenarios such as for Indic-to-Indic Languages (IL-IL). Hence, this article is an attempt to address and develop the baseline models for low-resource languages i.e., IL-IL (for 11 Indic Languages (ILs)) in a multilingual environment. The models are built on the Samanantar corpus and analyzed on the Flores-200 corpus. All the models are evaluated using standard evaluation metrics i.e., Bilingual Evaluation Understudy (BLEU) score (with the range of 0 to 100). This article examines the effect of the grouping of related languages, namely, East Indo-Aryan (EI), Dravidian (DR), and West Indo-Aryan (WI) on the MNMT model. From the experiments, the results reveal that related language grouping is beneficial for the WI group only while it is detrimental for the EI group and it shows an inconclusive effect on the DR group. The role of pivot-based MNMT models in enhancing translation quality is also investigated in this article. Owing to the presence of large good-quality corpora from English (EN) to ILs, MNMT IL-IL models using EN as a pivot are built and examined. To achieve this, English-Indic Language (EN-IL) models are developed with and without the usage of related languages. Results show that the use of related language grouping is advantageous specifically for EN to ILs. Thus, related language groups are used for the development of pivot MNMT models. It is also observed that the usage of pivot models greatly improves MNMT baselines. Furthermore, the effect of transliteration on ILs is also analyzed in this article. To explore transliteration, the best MNMT models from the previous approaches (in most of cases pivot model using related groups) are determined and built on corpus transliterated from the corresponding scripts to a modified Indian language Transliteration script (ITRANS). The outcome of the experiments indicates that transliteration helps the models built for lexically rich languages, with the best increment of BLEU scores observed in Malayalam (ML) and Tamil (TA), i.e., 6.74 and 4.72, respectively. The BLEU score using transliteration models ranges from 7.03 to 24.29. The best model obtained is the Punjabi (PA)-Hindi (HI) language pair trained on PA-WI transliterated corpus.
引用
收藏
页数:32
相关论文
共 57 条
[1]  
Aharoni R, 2019, Arxiv, DOI arXiv:1903.00089
[2]  
[Anonymous], 2017, Transactions of the Association for Computational Linguistics, DOI DOI 10.1162/TACL_A_00065
[3]  
Baruah R., 2021, Trans. Asian Low-resour. Lang. Inf. Process., V21, P1
[4]  
Baruah Rupjyoti, 2020, P 7 WORKSH AS TRANSL, P118
[5]  
Beekes Robert S. P., 2011, Compar. Indo-Eur, P1
[6]   Morphology to Molecular Anthropology: Castes and Tribes of India [J].
Bhasin, M. K. .
INTERNATIONAL JOURNAL OF HUMAN GENETICS, 2009, 9 (3-4) :145-230
[7]  
Blackwood G, 2018, Arxiv, DOI arXiv:1806.03280
[8]  
Codes for the representation of names of languages, 2023, ISO 639-3:2007
[9]  
Costa-jussa M. R., 2022, arXiv
[10]   Deep Learning-based POS Tagger and Chunker for Odia Language Using Pre-trained Transformers [J].
Dalai, Tusarkanta ;
Kumarmishra, Tapas ;
Sa, Andpankaj K. .
ACM TRANSACTIONS ON ASIAN AND LOW-RESOURCE LANGUAGE INFORMATION PROCESSING, 2024, 23 (02)