Convex-Concave Programming: An Effective Alternative for Optimizing Shallow Neural Networks

In this study, we address the challenges of non-convex optimization in neural networks (NNs) by formulating the training of multilayer perceptron (MLP) NNs as a difference of convex functions (DC) problem. Utilizing the basic convex-concave algorithm to solve our DC problems, we introduce two alternative optimization techniques, DC-GD and DC-OPT , for determining MLP parameters. By leveraging the non-uniqueness property of the convex components in DC functions, we generate strongly convex components for the DC NN cost function. This strong convexity enables our proposed algorithms, DC-GD and DC-OPT , to achieve an iteration complexity of O(log(1/epsilon)) , surpassing that of other solvers, such as stochastic gradient descent ( SGD ), which has an iteration complexity of O(1/epsilon) . This improvement raises the convergence rate from sublinear ( SGD ) to linear (ours) while maintaining comparable total computational costs . Furthermore, conventional NN optimizers like SGD , RMSprop , and Adam are highly sensitive to the learning rate, adding computational overhead for practitioners in selecting an appropriate learning rate. In contrast, our DC-OPT algorithm is hyperparameter-free (i.e., it requires no learning rate), and our DC-GD algorithm is less sensitive to the learning rate, offering comparable accuracy to other solvers. Additionally, we extend our approach to a convolutional NN architecture, demonstrating its applicability to modern NNs. We evaluate the performance of our proposed algorithms by comparing them to conventional optimizers such as SGD , RMSprop , and Adam across various test cases. The results suggest that our approach is a viable alternative for optimizing shallow MLP NNs.