A Lightweight Transformer with Convolutional Attention

被引:0
|
作者
Zeng, Kungan [1 ]
Paik, Incheon [1 ]
机构
[1] Univ Aizu, Sch Comp Sci & Engn, Fukushima, Japan
来源
2020 11TH INTERNATIONAL CONFERENCE ON AWARENESS SCIENCE AND TECHNOLOGY (ICAST) | 2020年
关键词
neural machine translation; Transformer; CNN; Muti-head attention;
D O I
10.1109/ICAST51195.2020.9319489
中图分类号
TM [电工技术]; TN [电子技术、通信技术];
学科分类号
0808 ; 0809 ;
摘要
Neural machine translation (NMT) goes through rapid development because of the application of various deep learning techs. Especially, how to construct a more effective structure of NMT attracts more and more attention. Transformer is a state-of-the-art architecture in NMT. It replies on the self-attention mechanism exactly instead of recurrent neural networks (RNN). The Multi-head attention is a crucial part that implements the self-attention mechanism, and it also dramatically affects the scale of the model. In this paper, we present a new Multi-head attention by combining convolution operation. In comparison with the base Transformer, our approach can reduce the number of parameters effectively. And we perform a reasoned experiment. The result shows that the performance of the new model is similar to the base model.
引用
收藏
页数:6
相关论文
共 50 条
  • [1] A lightweight convolutional swin transformer with cutmix augmentation and CBAM attention for compound emotion recognition
    Nidhi
    Verma, Bindu
    APPLIED INTELLIGENCE, 2024, : 7793 - 7809
  • [2] Efficient Lightweight Image Denoising with Triple Attention Transformer
    Zhou, Yubo
    Lin, Jin
    Ye, Fangchen
    Qu, Yanyun
    Xie, Yuan
    THIRTY-EIGHTH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE, VOL 38 NO 7, 2024, : 7704 - 7712
  • [3] Transformer with difference convolutional network for lightweight universal boundary detection
    Li, Mingchun
    Liu, Yang
    Chen, Dali
    Chen, Liangsheng
    Liu, Shixin
    PLOS ONE, 2024, 19 (04):
  • [4] Fully Convolutional Transformer with Local-Global Attention
    Lee, Sihaeng
    Yi, Eojindl
    Lee, Janghyeon
    Yoo, Jinsu
    Lee, Honglak
    Kim, Seung Hwan
    2022 IEEE/RSJ INTERNATIONAL CONFERENCE ON INTELLIGENT ROBOTS AND SYSTEMS (IROS), 2022, : 552 - 559
  • [5] CATNet: Convolutional attention and transformer for monocular depth estimation
    Tang, Shuai
    Lu, Tongwei
    Liu, Xuanxuan
    Zhou, Huabing
    Zhang, Yanduo
    PATTERN RECOGNITION, 2024, 145
  • [6] A lightweight transformer with linear self-attention for defect recognition
    Zhai, Yuwen
    Li, Xinyu
    Gao, Liang
    Gao, Yiping
    ELECTRONICS LETTERS, 2024, 60 (17)
  • [7] Lightweight and Efficient Human Pose Estimation Fusing Transformer and Attention
    Wu, Chengpeng
    Tan, Guangxing
    Chen, Haifeng
    Li, Chunyu
    Computer Engineering and Applications, 2024, 60 (22) : 197 - 208
  • [8] Finger vein recognition based on lightweight convolutional attention model
    Zhang, Zhongxia
    Wang, Mingwen
    IET IMAGE PROCESSING, 2023, 17 (06) : 1864 - 1873
  • [9] Lightweight Facial Expression Recognition Method Based on Convolutional Attention
    Yin Pengbo
    Pan Weimin
    Zhang Haijun
    LASER & OPTOELECTRONICS PROGRESS, 2021, 58 (12)
  • [10] ConTrans: Improving Transformer with Convolutional Attention for Medical Image Segmentation
    Lin, Ailiang
    Xu, Jiayu
    Li, Jinxing
    Lu, Guangming
    MEDICAL IMAGE COMPUTING AND COMPUTER ASSISTED INTERVENTION, MICCAI 2022, PT V, 2022, 13435 : 297 - 307