Momentum Acceleration of Quasi-Newton Training for Neural Networks

被引:5
作者
Mahboubi, Shahrzad [1 ]
Indrapriyadarsini, S. [2 ]
Ninomiya, Hiroshi [1 ]
Asai, Hideki [2 ]
机构
[1] Shonan Inst Technol, 1-1-25 Tsujido Nishikaigan, Fujisawa, Kanagawa 2518511, Japan
[2] Shizuoka Univ, Naka Ku, 3-5-1 Johoku, Hamamatsu, Shizuoka 4328011, Japan
来源
PRICAI 2019: TRENDS IN ARTIFICIAL INTELLIGENCE, PT II | 2019年 / 11671卷
关键词
Neural networks; Training algorithm; Quasi-Newton method; Nesterov's accelerated quasi-Newton method; Momentum terms;
D O I
10.1007/978-3-030-29911-8_21
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
This paper describes a novel acceleration technique of quasi-Newton method (QN) using momentum terms for training in neural networks. Recently, Nesterov's accelerated quasi-Newton method (NAQ) has shown that the momentum term is effective in reducing the number of iterations and in accelerating its convergence speed. However, the gradients had to calculate two times during one iteration in the NAQ training. This increased the computation time of a training loop compared with the conventional QN. In this research, an improvement to NAQ is done by approximating the Nesterov's accelerated gradient used in NAQ as a linear combination of the current and previous gradients. Then the gradient is calculated only once per iteration same as QN. The performance of the proposed algorithm is evaluated through computer simulations on a benchmark problem of the function modeling and real-world problems of the microwave circuit modeling. The results show the significant acceleration in the computation time compared with conventional training algorithms.
引用
收藏
页码:268 / 281
页数:14
相关论文
共 50 条
[41]   A quasi-Newton type method for equilibrium problems [J].
Sousa, Leonardo A. ;
Scheimberg, Susana ;
Santos, Pedro Jorge S. ;
Santos, Paulo Sergio M. .
NUMERICAL ALGORITHMS, 2022, 89 (03) :1129-1143
[42]   Newton and quasi-Newton methods for a class of nonsmooth equations and related problems [J].
Sun, DF ;
Han, JY .
SIAM JOURNAL ON OPTIMIZATION, 1997, 7 (02) :463-480
[43]   Quasi-Newton acceleration of ILU preconditioners for nonlinear two-phase flow equations in porous media [J].
Bergamaschi, L. ;
Bru, R. ;
Martinez, A. ;
Putti, M. .
ADVANCES IN ENGINEERING SOFTWARE, 2012, 46 (01) :63-68
[45]   A QUASI-NEWTON BUNDLE METHOD BASED ON APPROXIMATE SUBGRADIENTS [J].
Shen Jie ;
Pang Li-Ping .
JOURNAL OF APPLIED MATHEMATICS AND COMPUTING, 2007, 23 (1-2) :361-367
[46]   A new regularized quasi-Newton algorithm for unconstrained optimization [J].
Zhang, Hao ;
Ni, Qin .
APPLIED MATHEMATICS AND COMPUTATION, 2015, 259 :460-469
[47]   THE QUASI-NEWTON METHOD FOR THE COMPOSITE MULTIOBJECTIVE OPTIMIZATION PROBLEMS [J].
Peng, Jianwen ;
Zhang, Xue-Qing ;
Zhang, Tao .
JOURNAL OF NONLINEAR AND CONVEX ANALYSIS, 2024, 25 (10) :2557-2569
[48]   New quasi-Newton methods for unconstrained optimization problems [J].
Wei, Zengxin ;
Li, Guoyin ;
Qi, Liqun .
APPLIED MATHEMATICS AND COMPUTATION, 2006, 175 (02) :1156-1188
[49]   Practical Quasi-Newton algorithms for singular nonlinear systems [J].
Sandra Buhmiler ;
Nataša Krejić ;
Zorana Lužanin .
Numerical Algorithms, 2010, 55 :481-502
[50]   A quasi-Newton bundle method based on approximate subgradients [J].
Shen Jie ;
Pang Li-Ping .
Journal of Applied Mathematics and Computing, 2007, 23 (1-2) :361-367