Pruning Deep Neural Networks for Green Energy-Efficient Models: A Survey

被引:5
作者
Tmamna, Jihene [1 ]
Ben Ayed, Emna [1 ,2 ]
Fourati, Rahma [1 ,3 ]
Gogate, Mandar [4 ]
Arslan, Tughrul [5 ]
Hussain, Amir [4 ]
Ayed, Mounir Ben [1 ,6 ]
机构
[1] Univ Sfax, Natl Engn Sch Sfax ENIS, Res Grp Intelligent Machines, BP 1173, Sfax 3038, Tunisia
[2] Polytech Sfax IPSAS, Ind Res Lab 4 0, Ave 5 August,Rue Said Aboubaker, Sfax 3002, Tunisia
[3] Univ Jendouba, Fac Sci Jurid Econ & Gest Jendouba, Jendouba 8189, Tunisia
[4] Edinburgh Napier Univ, Sch Comp, Merchiston Campus, Edinburgh EH10 5DT, Scotland
[5] Sch Comp Engn & Built Environm, Edinburgh EH9 3FF, Scotland
[6] Univ Sfax, Fac Sci Sfax, Comp Sci & Commun Dept, Sfax, Tunisia
基金
英国工程与自然科学研究理事会;
关键词
Deep convolutional neural networks; Green deep learning; Neural network compression; Neural network pruning; ARCHITECTURES; COMPRESSION; RELEVANCE; FRAMEWORK; GRADIENT;
D O I
10.1007/s12559-024-10313-0
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Over the past few years, larger and deeper neural network models, particularly convolutional neural networks (CNNs), have consistently advanced state-of-the-art performance across various disciplines. Yet, the computational demands of these models have escalated exponentially. Intensive computations hinder not only research inclusiveness and deployment on resource-constrained devices, such as Edge Internet of Things (IoT) devices, but also result in a substantial carbon footprint. Green deep learning has emerged as a research field that emphasizes energy consumption and carbon emissions during model training and inference, aiming to innovate with light and energy-efficient neural networks. Various techniques are available to achieve this goal. Studies show that conventional deep models often contain redundant parameters that do not alter outcomes significantly, underpinning the theoretical basis for model pruning. Consequently, this timely review paper seeks to systematically summarize recent breakthroughs in CNN pruning methods, offering necessary background knowledge for researchers in this interdisciplinary domain. Secondly, we spotlight the challenges of current model pruning methods to inform future avenues of research. Additionally, the survey highlights the pressing need for the development of innovative metrics to effectively balance diverse pruning objectives. Lastly, it investigates pruning techniques oriented towards sophisticated deep learning models, including hybrid feedforward CNNs and long short-term memory (LSTM) recurrent neural networks, a field ripe for exploration within green deep learning research.
引用
收藏
页码:2931 / 2952
页数:22
相关论文
共 177 条
[1]   Unlocking the Potential of Two-Point Cells for Energy-Efficient and Resilient Training of Deep Nets [J].
Adeel, Ahsan ;
Adetomi, Adewale ;
Ahmed, Khubaib ;
Hussain, Amir ;
Arslan, Tughrul ;
Phillips, W. A. .
IEEE TRANSACTIONS ON EMERGING TOPICS IN COMPUTATIONAL INTELLIGENCE, 2023, 7 (03) :818-828
[2]   Development of a compressed FCN architecture for semantic segmentation using Particle Swarm Optimization [J].
Agarwal, Mohit ;
Gupta, Suneet K. ;
Biswas, K. K. .
NEURAL COMPUTING & APPLICATIONS, 2023, 35 (16) :11833-11846
[3]   Variational Information Distillation for Knowledge Transfer [J].
Ahn, Sungsoo ;
Hu, Shell Xu ;
Damianou, Andreas ;
Lawrence, Neil D. ;
Dai, Zhenwen .
2019 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2019), 2019, :9155-9163
[4]   Gradual Channel Pruning While Training Using Feature Relevance Scores for Convolutional Neural Networks [J].
Aketi, Sai Aparna ;
Roy, Sourjya ;
Raghunathan, Anand ;
Roy, Kaushik .
IEEE ACCESS, 2020, 8 :171924-171932
[5]   Group L1/2 Regularization for Pruning Hidden Layer Nodes of Feedforward Neural Networks [J].
Alemu, Habtamu Zegeye ;
Zhao, Junhong ;
Li, Feng ;
Wu, Wei .
IEEE ACCESS, 2019, 7 :9540-9557
[6]  
Allen-Zhu Z, 2019, ADV NEUR IN, V32
[7]   Deep compression of convolutional neural networks with low-rank approximation [J].
Astrid, Marcella ;
Lee, Seung-Ik .
ETRI JOURNAL, 2018, 40 (04) :421-434
[8]   Redundant feature pruning for accelerated inference in deep neural networks [J].
Ayinde, Babajide O. ;
Inanc, Tamer ;
Zurada, Jacek M. .
NEURAL NETWORKS, 2019, 118 :148-158
[9]  
Banner Ron, 2018, ADV NEURAL INFORM PR, V31
[10]  
Bochkovskiy A, 2020, Arxiv, DOI [arXiv:2004.10934, 10.48550/arXiv.2004.10934]