深度学习中注意力机制研究进展

被引:49
作者
刘建伟
刘俊文
罗雄麟
机构
[1] 中国石油大学(北京)自动化系
关键词
注意力机制; 全局/局部注意力机制; 硬/软注意力机制; 自注意力机制; 可解释性;
D O I
10.13374/j.issn2095-9389.2021.01.30.005
中图分类号
TP391.2 [翻译机]; TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
对注意力机制的主流模型进行了全面系统的概述.注意力机制模拟人类视觉选择性的机制,其核心的目的是从冗杂的信息中选择出对当前任务目标关联性更大、更关键的信息而过滤噪声,也就是高效率信息选择和关注机制.首先简要介绍和定义了注意力机制的原型,接着按照多个层面对各种注意力机制结构进行分类,然后对注意力机制的可解释性进行了阐述同时总结了在各种领域的应用,最后指出了注意力机制未来的发展方向以及会面临的挑战.
引用
收藏
页码:1499 / 1511
页数:13
相关论文
共 9 条
[1]   Top-Down Neural Attention by Excitation Backprop [J].
Zhang, Jianming ;
Bargal, Sarah Adel ;
Lin, Zhe ;
Brandt, Jonathan ;
Shen, Xiaohui ;
Sclaroff, Stan .
INTERNATIONAL JOURNAL OF COMPUTER VISION, 2018, 126 (10) :1084-1102
[2]  
ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs[J] . Wenpeng Yin,Hinrich Schütze,Bing Xiang,Bowen Zhou.Transactions of the Association for Computational Linguistics . 2016
[3]  
Action Recognition using Visual Attention[J] . Shikhar Sharma,Ryan Kiros,Ruslan Salakhutdinov.CoRR . 2015
[4]  
Visual attention: The past 25 years[J] . Marisa Carrasco.Vision Research . 2011 (13)
[5]  
Selective visual attention enables learning and recognition of multiple objects in cluttered scenes[J] . Dirk Walther,Ueli Rutishauser,Christof Koch,Pietro Perona.Computer Vision and Image Understanding . 2004 (1)
[6]  
Bi-directional block selfattention for fast and memory-efficient sequence modeling .2 Shen T,Zhou T Y,Long G D,et al. Proceedings of the 6th International Conference on Learning Representations . 2018
[7]  
Self-attention generative adversarial networks .2 Zhang H,Goodfellow I,Metaxas D,et al. Proceedings of the 36nd International Conference on Machine Learning . 2019
[8]  
Tree-structured attention with hierarchical accumulation .2 Nguyen X P,Joty S,Hoi S C H,et al. https://arxiv.org/abs/2002.08046 . 2020
[9]  
Convolutional Sequence to Sequence Learning .2 Gehring J,Auli M,Grangier D,et al. International Conference on Machine Learning . 2017