Enhanced mechanisms of pooling and channel attention for deep learning feature maps

被引:2
作者
Li, Hengyi [1 ]
Yue, Xuebin [1 ]
Meng, Lin [2 ]
机构
[1] Ritsumeikan Univ, Grad Sch Sci & Engn, Kusatsu, Shiga, Japan
[2] Ritsumeikan Univ, Coll Sci & Engn, Kusatsu, Shiga, Japan
关键词
DNNs; Max pooling; Average pooling; FMAPooling; Self-attention; FMAttn;
D O I
10.7717/peerj-cs.1161
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
The pooling function is vital for deep neural networks (DNNs). The operation is to generalize the representation of feature maps and progressively cut down the spatial size of feature maps to optimize the computing consumption of the network. Furthermore, the function is also the basis for the computer vision attention mechanism. However, as a matter of fact, pooling is a down-sampling operation, which makes the feature-map representation approximately to small translations with the summary statistic of adjacent pixels. As a result, the function inevitably leads to information loss more or less. In this article, we propose a fused max-average pooling (FMAPooling) operation as well as an improved channel attention mechanism (FMAttn) by utilizing the two pooling functions to enhance the feature representation for DNNs. Basically, the methods are to enhance multiple-level features extracted by max pooling and average pooling respectively. The effectiveness of the proposals is verified with VGG, ResNet, and MobileNetV2 architectures on CIFAR10/100 and ImageNet100. According to the experimental results, the FMAPooling brings up to 1.63% accuracy improvement compared with the baseline model; the FMAttn achieves up to 2.21% accuracy improvement compared with the previous channel attention mechanism. Furthermore, the proposals are extensible and could be embedded into various DNN models easily, or take the place of certain structures of DNNs. The computation burden introduced by the proposals is negligible.
引用
收藏
页数:18
相关论文
共 50 条
[41]   Deep Multi-Instance Learning with Induced Self-Attention for Medical Image Classification [J].
Li, Zhenliang ;
Yuan, Liming ;
Xu, Haixia ;
Cheng, Rui ;
Wen, Xianbin .
2020 IEEE INTERNATIONAL CONFERENCE ON BIOINFORMATICS AND BIOMEDICINE, 2020, :446-450
[42]   DenoiseNet: Deep Generator and Discriminator Learning Network With Self-Attention Applied to Ocean Data [J].
Mao, Mingqiu ;
Wang, Huajun ;
Nie, Peng ;
Xiao, Shipeng ;
Wu, Ruijie .
IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, 2022, 60
[43]   NON-LOCAL SELF-ATTENTION STRUCTURE FOR FUNCTION APPROXIMATION IN DEEP REINFORCEMENT LEARNING [J].
Wang, Zhixiang ;
Xiao, Xi ;
Hu, Guangwu ;
Yao, Yao ;
Zhang, Dianyan ;
Peng, Zhendong ;
Li, Qing ;
Xia, Shutao .
2019 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), 2019, :3042-3046
[44]   Deep Learning-Based Image Retrieval System with Clustering on Attention-Based Representations [J].
Rao S.S. ;
Ikram S. ;
Ramesh P. .
SN Computer Science, 2021, 2 (3)
[45]   Gridding and filtering method of gravity and magnetic data based on self-attention deep learning [J].
Ma G. ;
Wang Z. ;
Li L. .
Shiyou Diqiu Wuli Kantan/Oil Geophysical Prospecting, 2022, 57 (01) :34-42
[46]   Stochastic Economic Lot Scheduling via Self-Attention Based Deep Reinforcement Learning [J].
Song, Wen ;
Mi, Nan ;
Li, Qiqiang ;
Zhuang, Jing ;
Cao, Zhiguang .
IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING, 2024, 21 (02) :1457-1468
[47]   AttnPep: A Self-Attention-Based Deep Learning Method for Peptide Identification in Shotgun Proteomics [J].
Li, Yulin ;
He, Qingzu ;
Guo, Huan ;
Shuai, Stella C. ;
Cheng, Jinyan ;
Liu, Liyu ;
Shuai, Jianwei .
JOURNAL OF PROTEOME RESEARCH, 2024, 23 (02) :834-843
[48]   A deep learning sequence model based on self-attention and convolution for wind power prediction [J].
Liu, Chien-Liang ;
Chang, Tzu-Yu ;
Yang, Jie-Si ;
Huang, Kai-Bin .
RENEWABLE ENERGY, 2023, 219
[49]   DySAT: Deep Neural Representation Learning on Dynamic Graphs via Self-Attention Networks [J].
Sankar, Aravind ;
Wu, Yanhong ;
Gou, Liang ;
Zhang, Wei ;
Yang, Hao .
PROCEEDINGS OF THE 13TH INTERNATIONAL CONFERENCE ON WEB SEARCH AND DATA MINING (WSDM '20), 2020, :519-527
[50]   A Context-Aware Enhanced GRU Network With Feature-Temporal Attention for Prediction of Silicon Content in Hot Metal [J].
Li, Junfang ;
Yang, Chunjie ;
Li, Yuxuan ;
Xie, Shujia .
IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, 2022, 18 (10) :6631-6641