A Dynamic Layer-Wise Gradient Sparsity and Gradient Merging Optimization Method for Deep Neural Networks

A dynamic layer-wise gradient sparsity and gradient aggregation optimization strategy for deep neural net-works is proposed to address the challenges posed by substantial communication overhead, prolonged training duration, and suboptimal resource utilization associated -with the acceleration of large-scale deep neural net-works through data parallelism. Initially, a dynamic layer-wise gradient sparsity optimization method is proposed by combining gradient sparsity compression with pipeline parallelism. Each neural net-work layer is assigned an appropriate threshold, -which is adjusted dynamically in subsequent iterations to achieve adaptive compression of gradient transmission for each layer. Subsequently, a layer-wise gradient merging method is introduced. Leveraging dynamic programming, this method optimizes communication overhead, sparsity, and layer gradient computation time during layer-wise gradient merging, determining the optimal combination for merging multiple layers of small-scale gradient tensors into a single communication layer. This aims to reduce the high communication latency introduced during layer-wise gradient decision-making. Finally, the determined optimal layer-wise gradient merging combination is applied to the specific training iteration process. Experimental results demonstrate that the proposed method, compared to existing methods, significantly reduces communication overhead and enhances model training speed while ensuring model training accuracy. It achieves a maximum training speed up of 1. 99 times compared to the uncompressed method. © 2024 Xi'an Jiaotong University. All rights reserved.