Accelerated Primal-Dual Algorithm for Distributed Non-convex Optimization

This paper investigates accelerating the convergence of distributed optimization algorithms on non-convex problems. We propose a distributed stochastic gradient primal-dual with fixed parameters and "powerball" (DSGPA-F-PB) method to accelerate. We show that the proposed algorithm achieves the linear speedup convergence rate O(1/root nT) for general smooth (possibly non-convex) cost functions. We demonstrate the efficiency of the algorithm through numerical experiments by training two-layer fully connected neural networks and convolutional neural networks on the MNIST dataset to compare with state-of-the-art distributed stochastic gradient descent (SGD) algorithms and centralized SGD algorithms.