Reinforcement of BERT with Dependency-Parsing Based Attention Mask

被引：0

作者：

Mechouma, Toufik ^{[1
]}

Biskri, Ismail ^{[2
]}

Meunier, Jean Guy ^{[1
]}

机构：

[1] Univ Quebec Montreal, Montreal, PQ, Canada

[2] Univ Quebec Trois Rivieres, Trois Rivieres, PQ, Canada

来源：

ADVANCES IN COMPUTATIONAL COLLECTIVE INTELLIGENCE, ICCCI 2022 | 2022年 / 1653卷

关键词：

Bert; Transformers; Attention mechanisms; Dependency parsing;

D O I：

10.1007/978-3-031-16210-7_9

中图分类号：

TP18 [人工智能理论];

学科分类号：

081104 ; 0812 ; 0835 ; 1405 ;

摘要：

Dot-Product based attention mechanism is among recent attention mechanisms. It showed an outstanding performance with BERT. In this paper, we propose a dependency-parsing mask to reinforce the padding mask, at the multi-head attention units. Padding mask, is already used to filter padding positions. The proposed mask, aims to improve BERT attention filter. The conducted experiments, show that BERT performs better with the proposed mask.

引用

页码：112 / 122

页数：11

共 50 条

[31] Exploring Automatic Feature Selection for Transition-Based Dependency Parsing
Ballesteros, Miguel
PROCESAMIENTO DEL LENGUAJE NATURAL, 2013, (51): : 119 - 126
[32] A survey of syntactic-semantic parsing based on constituent and dependency structures
MeiShan Zhang
Science China Technological Sciences, 2020, 63 : 1898 - 1920
[33] Research on Methods of Microblogging Sentiment Feature Extraction Based on Dependency Parsing
Li Yonggan
Zhou Xueguang
Guo Wei
Zhang Huanguo
PROCEEDINGS OF THE 2015 INTERNATIONAL CONFERENCE ON INTELLIGENT SYSTEMS RESEARCH AND MECHATRONICS ENGINEERING, 2015, 121 : 581 - 589
[34] Design and Implementation of Weibo Sentiment Analysis Based on LDA and Dependency Parsing
Yonggan Li
Xueguang Zhou
Yan Sun
Huanguo Zhang
中国通信, 2016, 13 (11) : 91 - 105
[35] ADPG: Biomedical entity recognition based on Automatic Dependency Parsing Graph
Yang, Yumeng
Lin, Hongfei
Yang, Zhihao
Zhang, Yijia
Zhao, Di
Huai, Shuaiheng
JOURNAL OF BIOMEDICAL INFORMATICS, 2023, 140
[36] Mining Feature-Opinion from Reviews Based on Dependency Parsing
He, Tieke
Hao, Rui
Qi, Hang
Liu, Jia
Wu, Qing
INTERNATIONAL JOURNAL OF SOFTWARE ENGINEERING AND KNOWLEDGE ENGINEERING, 2016, 26 (9-10) : 1581 - 1591
[37] Design and Implementation of Weibo Sentiment Analysis Based on LDA and Dependency Parsing
Li, Yonggan
Zhou, Xueguang
Sun, Yan
Zhang, Huanguo
CHINA COMMUNICATIONS, 2016, 13 (11) : 91 - 105
[38] Combining Dependency Parsing and a Lexical Network Based on Lexical Functions for the Identification of Collocations
Fonseca, Alexsandro
Sadat, Fatiha
Lareau, Francois
COMPUTATIONAL AND CORPUS-BASED PHRASEOLOGY, EUROPHRAS 2017, 2017, 10596 : 447 - 461
[39] Deep Graph-Based Character-Level Chinese Dependency Parsing
Wu, Linzhi
Zhang, Meishan
IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING, 2021, 29 : 1329 - 1339
[40] A Multiscale Interactive Attention Short Text Classification Model Based on BERT
Zhou, Lu
Wang, Peng
Zhang, Huijun
Wu, Shengbo
Zhang, Tao
IEEE ACCESS, 2024, 12 : 160992 - 161001

← 1 2 3 4 5 →