DEEPAM: Toward Deeper Attention Module in Residual Convolutional Neural Networks

被引:1
|
作者
Zhong, Shanshan [1 ]
Wen, Wushao [1 ]
Qin, Jinghui [2 ]
Huang, Zhongzhan [1 ]
机构
[1] Sun Yat Sen Univ, Guangzhou, Peoples R China
[2] Guangdong Univ Technol, Guangzhou, Peoples R China
来源
ARTIFICIAL NEURAL NETWORKS AND MACHINE LEARNING-ICANN 2024, PT I | 2024年 / 15016卷
基金
中国国家自然科学基金;
关键词
Attention mechanism; Visual recognition;
D O I
10.1007/978-3-031-72332-2_26
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
The efficacy of depth in boosting the performance of residual convolutional neural networks (CNNs) has been well-established through abundant empirical or theoretical evidences. However, despite the attention module (AM) being a crucial component for high-performance CNNs, most existing research primarily focuses on their structural design, overlooking a direct investigation into the impact of AM depth on performance. Therefore, in this paper, we explore the influence of AM depth under various settings in detail. We observe that (1) appropriately increasing AM depth significantly boosts performance; (2) deepening AM exhibits a higher cost-effectiveness compared to traditional backbone deepening. However, deepening AM introduces inherent challenges in terms of parameter and inference cost. To mitigate them while enjoying the benefit of deepening AM, we propose a novel AM called DEEPAM, leveraging mechanisms from recurrent neural networks and the design of lightweight AMs. Extensive experiments on widely-used benchmarks and popular attention networks validate the effectiveness of our proposed DEEPAM.
引用
收藏
页码:405 / 418
页数:14
相关论文
共 50 条
  • [1] An Attention Module for Convolutional Neural Networks
    Zhu, Baozhou
    Hofstee, Peter
    Lee, Jinho
    Al-Ars, Zaid
    ARTIFICIAL NEURAL NETWORKS AND MACHINE LEARNING - ICANN 2021, PT I, 2021, 12891 : 167 - 178
  • [2] A New Cyclic Spatial Attention Module for Convolutional Neural Networks
    Li Daihui
    Zeng Shangyou
    Li Wenhui
    Yang Lei
    2019 IEEE 11TH INTERNATIONAL CONFERENCE ON COMMUNICATION SOFTWARE AND NETWORKS (ICCSN 2019), 2019, : 607 - 611
  • [3] A Simple and Light-Weight Attention Module for Convolutional Neural Networks
    Park, Jongchan
    Woo, Sanghyun
    Lee, Joon-Young
    Kweon, In So
    INTERNATIONAL JOURNAL OF COMPUTER VISION, 2020, 128 (04) : 783 - 798
  • [4] A Simple and Light-Weight Attention Module for Convolutional Neural Networks
    Jongchan Park
    Sanghyun Woo
    Joon-Young Lee
    In So Kweon
    International Journal of Computer Vision, 2020, 128 : 783 - 798
  • [5] Image Denoising using Attention-Residual Convolutional Neural Networks
    Pires, Rafael G.
    Santos, Daniel F. S.
    Santos, Claudio F. G.
    Santana, Marcos C. S.
    Papa, Joao P.
    2020 33RD SIBGRAPI CONFERENCE ON GRAPHICS, PATTERNS AND IMAGES (SIBGRAPI 2020), 2020, : 101 - 107
  • [6] Deeper Depth Prediction with Fully Convolutional Residual Networks
    Laina, Iro
    Rupprecht, Christian
    Belagiannis, Vasileios
    Tombari, Federico
    Navab, Nassir
    PROCEEDINGS OF 2016 FOURTH INTERNATIONAL CONFERENCE ON 3D VISION (3DV), 2016, : 239 - 248
  • [7] FDAM: full-dimension attention module for deep convolutional neural networks
    Cai, Silin
    Wang, Changping
    Ding, Jiajun
    Yu, Jun
    Fan, Jianping
    INTERNATIONAL JOURNAL OF MULTIMEDIA INFORMATION RETRIEVAL, 2022, 11 (04) : 599 - 610
  • [8] Semantic Face Segmentation Using Convolutional Neural Networks With a Supervised Attention Module
    Hizukuri, Akiyoshi
    Hirata, Yuto
    Nakayama, Ryohei
    IEEE ACCESS, 2023, 11 : 116892 - 116902
  • [9] SimAM: A Simple, Parameter-Free Attention Module for Convolutional Neural Networks
    Yang, Lingxiao
    Zhang, Ru-Yuan
    Li, Lida
    Xie, Xiaohua
    INTERNATIONAL CONFERENCE ON MACHINE LEARNING, VOL 139, 2021, 139
  • [10] FDAM: full-dimension attention module for deep convolutional neural networks
    Silin Cai
    Changping Wang
    Jiajun Ding
    Jun Yu
    Jianping Fan
    International Journal of Multimedia Information Retrieval, 2022, 11 : 599 - 610