Multi-stage transfer learning with BERTology-based language models for question answering system in vietnamese

With the fast growth of information science and engineering, a large number of textual data generated are valuable for natural language processing and its applications. Particularly, finding correct answers to natural language questions or queries requires spending tremendous time and effort in human life. While using search engines to discover information, users manually determine the answer to a given question on a range of retrieved texts or documents. Question answering relies heavily on the capability to automatically comprehend questions in human language and extract meaningful answers from a single text. In recent years, such question–answering systems have become increasingly popular using machine reading comprehension techniques. On the other hand, high-resource languages (e.g., English and Chinese) have witnessed tremendous growth in question-answering methodologies based on various knowledge sources. Besides, powerful BERTology-based language models only encode texts with a limited length. The longer texts contain more distractor sentences that affect the QA system performance. Vietnamese has a variety of question words in the same question type. To address these challenges, we propose ViQAS, a new question–answering system with multi-stage transfer learning using language models based on BERTology for a low-resource language such as Vietnamese. Last but not least, our QA system is integrated with Vietnamese characteristics and transformer-based evidence extraction techniques into an effective contextualized language model-based QA system. As a result, our proposed system outperforms our forty retriever-reader QA configurations and seven state-of-the-art QA systems such as DrQA, BERTserini, BERTBM25, XLMRQA, ORQA, COBERT, and NeuralQA on three Vietnamese benchmark question answering datasets.