Long text semantic matching model based on BERT and densecomposite network

被引：0
|
作者：
Chen Y.-L. [1 ]
Gao Z.-C. [1 ]
Cai X.-D. [2 ]
机构：
[1] School of Mechanical and Electrical Engineering, Guilin University of Electronic Technology, Guilin
[2] School of Information and Communication, Guilin University of Electronic Technology, Guilin
来源：
Jilin Daxue Xuebao (Gongxueban)/Journal of Jilin University (Engineering and Technology Edition) | 2024年 / 54卷 / 01期
关键词：
BERT; Bi-LSTM; deep learning; dense composite network; long text semantic matching; TextCNN;
D O I：
10.13229/j.cnki.jdxbgxb.20220239
中图分类号：
学科分类号：
摘要：
In the semantic matching of long texts，it is challenging to capture the before-and-after connections and topic information，which often results in poor semantic matching. This paper proposes a long text semantic matching method based on BERT and dense composite network. Through the dense connection of BERT embedding and composite network，the accuracy of long semantic matching is significantly improved. First，the sentence pair is input into the BERT pre-training model，and accurate word vector representation is obtained through iterative feedback，and then high-quality sentence pair semantic information is obtained. Secondly，a dense composite network is designed. Bi-LSTM first obtains the global semantic information of sentence pairs，and then TextCNN extracts and integrates local semantic information to obtain the key features of each sentence and the correspondence between sentence pairs，and the BERT Fusion with the hidden output of Bi-LSTM and the pooled output of TextCNN. Finally，summarizing the association state between networks during the training process can effectively prevent network degradation and enhance the model’s judgment ability. The experimental results show that on the community question answering（CQA）long text dataset，the method in this paper has a significant effect，with an average improvement of 45%. © 2024 Editorial Board of Jilin University. All rights reserved.
引用
收藏
页码：232 / 239
页数：7
相关论文
共 21 条
  • [11] Li Bo-han, Zhou Hao, He Jun-xian, Et al., On the sentence embeddings from pre-trained language models, The Conference on Empirical Methods in Natural Language Processing, pp. 9119-9130, (2020)
  • [12] Peinelt N, Nguyen D, Liakata M., tBERT: topic models and BERT joining forces for semantic similarity detection, Proceedings of the 58rd Annual Meeting of the Association for Computational Linguistics, pp. 7047-7055, (2020)
  • [13] Chen Han-jie, Zheng Guang-tao, Ji Yang-feng, Generating hierarchical explanations on text classification via feature interaction detection, Proceedings of the 58rd Annual Meeting of the Association for Computational Linguistics, pp. 5578-5593, (2020)
  • [14] Tsang M, Cheng D, Liu H, Et al., Feature interaction interpretability: a case for explaining ad-recommendation systems via neural interaction detection[J/OL]
  • [15] Gao J, He D, Tan X, Et al., Representation degeneration problem in training natural language generation models[J/OL]
  • [16] Ethayarajh K., How contextual are contextualized word representations? [C], Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pp. 55-65, (2019)
  • [17] Yan Yuan-meng, Li Ru-mei, Wang Si-rui, Et al., ConSERT: a contrastive framework for self-supervised sentence representation transfer[C], Association for Computational Linguistics and International Joint Conference on Natural Language Processing, pp. 5065-5075, (2021)
  • [18] Schick T, Schutze H., Generating datasets with pretrained language models[J/OL][2021-10-21]
  • [19] Chen Han-jie, Song Feng, Jatin Ganhotra, Et al., Explaining neural network predictions on sentence pairs via learning word-group masks, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics, pp. 3917-3930, (2021)
  • [20] Balazs Jorge A, Matsuo Y., Gating mechanisms for combining character and word-level word representations: an empirical study, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pp. 110-124, (2019)
123