Momentum Acceleration of Quasi-Newton Training for Neural Networks

This paper describes a novel acceleration technique of quasi-Newton method (QN) using momentum terms for training in neural networks. Recently, Nesterov's accelerated quasi-Newton method (NAQ) has shown that the momentum term is effective in reducing the number of iterations and in accelerating its convergence speed. However, the gradients had to calculate two times during one iteration in the NAQ training. This increased the computation time of a training loop compared with the conventional QN. In this research, an improvement to NAQ is done by approximating the Nesterov's accelerated gradient used in NAQ as a linear combination of the current and previous gradients. Then the gradient is calculated only once per iteration same as QN. The performance of the proposed algorithm is evaluated through computer simulations on a benchmark problem of the function modeling and real-world problems of the microwave circuit modeling. The results show the significant acceleration in the computation time compared with conventional training algorithms.