Reinforcement of BERT with Dependency-Parsing Based Attention Mask

被引：0

作者：

Mechouma, Toufik ^{[1
]}

Biskri, Ismail ^{[2
]}

Meunier, Jean Guy ^{[1
]}

机构：

[1] Univ Quebec Montreal, Montreal, PQ, Canada

[2] Univ Quebec Trois Rivieres, Trois Rivieres, PQ, Canada

来源：

ADVANCES IN COMPUTATIONAL COLLECTIVE INTELLIGENCE, ICCCI 2022 | 2022年 / 1653卷

关键词：

Bert; Transformers; Attention mechanisms; Dependency parsing;

D O I：

10.1007/978-3-031-16210-7_9

中图分类号：

TP18 [人工智能理论];

学科分类号：

081104 ; 0812 ; 0835 ; 1405 ;

摘要：

Dot-Product based attention mechanism is among recent attention mechanisms. It showed an outstanding performance with BERT. In this paper, we propose a dependency-parsing mask to reinforce the padding mask, at the multi-head attention units. Padding mask, is already used to filter padding positions. The proposed mask, aims to improve BERT attention filter. The conducted experiments, show that BERT performs better with the proposed mask.

引用

页码：112 / 122

页数：11

共 50 条

[1] Fine-Tuning BERT-Based Pre-Trained Models for Arabic Dependency Parsing
Al-Ghamdi, Sharefah
Al-Khalifa, Hend
Al-Salman, Abdulmalik
APPLIED SCIENCES-BASEL, 2023, 13 (07):
[2] SHORT-TEXT QUESTION CLASSIFICATION BASED ON DEPENDENCY PARSING AND ATTENTION MECHANISM
Fang, An
PROCEEDINGS OF 2019 INTERNATIONAL CONFERENCE ON MACHINE LEARNING AND CYBERNETICS (ICMLC), 2019, : 498 - 503
[3] BERT Probe: A python']python package for probing attention based robustness evaluation of BERT models
Khan, Shahrukh
Shahid, Mahnoor
Singh, Navdeeppal
SOFTWARE IMPACTS, 2022, 13
[4] Knowledge enhancement BERT based on domain dictionary mask
Cao, Xianglin
Xiao, Hong
Jiang, Wenchao
JOURNAL OF HIGH SPEED NETWORKS, 2023, 29 (02) : 121 - 128
[5] Improving Graph-Based Dependency Parsing Models With Dependency Language Models
Zhang, Min
Chen, Wenliang
Duan, Xiangyu
Zhang, Rong
IEEE TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING, 2013, 21 (11): : 2313 - 2323
[6] Relationship classification based on dependency parsing and the pretraining model
Baosheng Yin
Yifei Sun
Soft Computing, 2022, 26 : 8575 - 8583
[7] A Burmese Dependency Parsing Method Based on Transfer Learning
Mao, Cunli
Man, Zhibo
Yu, Zhengtao
Wang, Zhenhan
Gao, Shengxiang
Zhang, Yafei
2020 INTERNATIONAL CONFERENCE ON ASIAN LANGUAGE PROCESSING (IALP 2020), 2020, : 92 - 97
[8] The Chinese Open Relation Extraction Based on Dependency Parsing
Wang Yuzhao
Yang Yunfei
Zhao Ruixue
PROCEEDINGS OF THE 2017 5TH INTERNATIONAL CONFERENCE ON FRONTIERS OF MANUFACTURING SCIENCE AND MEASURING TECHNOLOGY (FMSMT 2017), 2017, 130 : 1212 - 1216
[9] Mining Product Reviews Based on Shallow Dependency Parsing
Zhang, Qi
Ogihara, Mitsunori
Wu, Yuanbin
Johnson, Joseph
Li, Tao
Huang, Xuanjing
PROCEEDINGS 32ND ANNUAL INTERNATIONAL ACM SIGIR CONFERENCE ON RESEARCH AND DEVELOPMENT IN INFORMATION RETRIEVAL, 2009, : 726 - 727
[10] Transition-Based Dependency Parsing Exploiting Supertags
Ouchi, Hiroki
Duh, Kevin
Shindo, Hiroyuki
Matsumoto, Yuji
IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING, 2016, 24 (11) : 2059 - 2068

← 1 2 3 4 5 →