Idiom Cloze Algorithm Integrating with Pre-trained Language Model

被引：0
作者：
Ju S.-G. [1 ]
Huang F.-Y. [1 ]
Sun J.-P. [1 ]
机构：
[1] College of Computer Science, Sichuan University, Chengdu
来源：
Ruan Jian Xue Bao/Journal of Software | 2022年 / 33卷 / 10期
关键词：
Chinese idiom cloze task; deep learning; pre-trained language model; text matching;
D O I：
10.13328/j.cnki.jos.006307
中图分类号：
学科分类号：
摘要：
One of the crucial tasks in the field of natural language processing (NLP) is identifying suitable idioms due to context. The available research considers the Chinese idiom cloze task as a textual similarity task. Although the current pre-trained language model plays an important role in textual similarity, it also has apparent defects. When pre-trained language model is used as a feature extractor, it ignores the mutual information between sentences; while as a text matcher, it requires high computational cost and long running time. In addition, the matching between context and candidate idioms is asymmetric, which influences the effect of the pre-trained language model as a text matcher. In order to solve the above two problems, this study is motivated by the idea of parameter sharing and proposes a TALBERT-blank network. Idiom selection is transformed from a context-based asymmetric matching process into a blank-based symmetric matching process by TALBERT-blank. The pre-trained language model acts as both a feature extractor and a text matcher, and the sentence vector is utilized for latent semantic matches. This greatly reduces the number of parameters and the consumption of memory, improves the speed of train and inference while maintaining accuracy, and produces a lightweight and efficient effect. The experimental results of this model on CHID data set prove that compared with ALBERT text matcher, the calculation time is further shortened by 54.35 percent for the compression model with a greater extent under the condition of maintaining accuracy. © 2022 Chinese Academy of Sciences. All rights reserved.
引用
收藏
页码：3793 / 3805
页数：12
相关论文
共 40 条
  • [1] Jiang Z, Zhang B, Huang L, Et al., Chengyu cloze test, Proc. of the 13th Workshop on Innovative Use of NLP for Building Educational Applications, pp. 154-158, (2018)
  • [2] Wang L, Yu S., Construction of Chinese idiom knowledge-base and its applications, Proc. of the 2010 Workshop on Multiword Expressions: From Theory to Applications, pp. 11-18, (2010)
  • [3] Shao Y, Sennrich R, Webber B, Et al., Evaluating machine translation performance on chinese idioms with a blacklist method, Proc. of the 11th Edition of the Language Resources and Evaluation Conf, pp. 31-38, (2018)
  • [4] Liu P, Qian K, Qiu X, Et al., Idiom-aware compositional distributed semantics, Proc. of the 2017 Conf. on Empirical Methods in Natural Language Processing, pp. 1204-1213, (2017)
  • [5] Zheng C, Huang M, Sun A., CHID: A large-scale Chinese idiom dataset for cloze test, Proc. of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 778-787, (2019)
  • [6] Wang M, Xiao M, Li C, Et al., STAC: Science toolkit based on Chinese idiom knowledge graph, Proc. of the Workshop on Extracting Structured Knowledge from Scientific Publications, pp. 57-61, (2019)
  • [7] Hochreiter S, Schmidhuber J., Long short-term memory, Neural Computation, 9, 8, pp. 1735-1780, (1997)
  • [8] Devlin J, Chang MW, Lee K, Et al., BERT: Pre-training of deep bidirectional transformers for language understanding, Proc. of the 2019 NAACL-HLT, 11, pp. 4171-4186, (2019)
  • [9] Cieslicka A., Literal salience in on-line processing of idiomatic expressions by second language learners, Second Language Research, 22, 2, pp. 115-144, (2006)
  • [10] Liu Y, Liu B, Shan L, Et al., Modelling context with neural networks for recommending idioms in essay writing, Neurocomputing, 275, pp. 2287-2293, (2018)
1234