A nonlinear conjugate gradient method with complexity guarantees and its application to nonconvex regression

被引:10
作者
Chan-Renous-Legoubin, Remi
Royer, Clement W. [1 ,2 ]
机构
[1] Univ Paris Dauphine PSL, F-75016 Paris, France
[2] Univ Paris Dauphine PSL, LAMSADE, CNRS, F-75016 Paris, France
关键词
LINE-SEARCH ALGORITHMS; DESCENT; REGULARIZATION; CG;
D O I
10.1016/j.ejco.2022.100044
中图分类号
C93 [管理学]; O22 [运筹学];
学科分类号
070105 ; 12 ; 1201 ; 1202 ; 120202 ;
摘要
Nonlinear conjugate gradients are among the most popular techniques for solving continuous optimization problems. Al-though these schemes have long been studied from a global convergence standpoint, their worst-case complexity proper-ties have yet to be fully understood, especially in the noncon-vex setting. In particular, it is unclear whether nonlinear con-jugate gradient methods possess better guarantees than first-order methods such as gradient descent. Meanwhile, recent experiments have shown impressive performance of standard nonlinear conjugate gradient techniques on certain nonconvex problems, even when compared with methods endowed with the best known complexity guarantees. In this paper, we propose a nonlinear conjugate gradient scheme based on a simple line-search paradigm and a modified restart condition. These two ingredients allow for monitoring the properties of the search directions, which is instrumental in obtaining complexity guarantees. Our complexity results il-lustrate the possible discrepancy between nonlinear conjugate gradient methods and classical gradient descent. A numerical investigation on nonconvex robust regression problems as well as a standard benchmark illustrate that the restarting condi-tion can track the behavior of a standard implementation. (c) 2022 The Author(s). Published by Elsevier Ltd on behalf of Association of European Operational Research Societies (EURO).
引用
收藏
页数:26
相关论文
共 36 条
  • [1] FITTING OF POWER-SERIES, MEANING POLYNOMIALS, ILLUSTRATED ON BAND-SPECTROSCOPIC DATA
    BEATON, AE
    TUKEY, JW
    [J]. TECHNOMETRICS, 1974, 16 (02) : 147 - 185
  • [2] THE USE OF QUADRATIC REGULARIZATION WITH A CUBIC DESCENT CONDITION FOR UNCONSTRAINED OPTIMIZATION
    Birgin, E. G.
    Martinez, J. M.
    [J]. SIAM JOURNAL ON OPTIMIZATION, 2017, 27 (02) : 1049 - 1074
  • [3] Worst-case evaluation complexity for unconstrained nonlinear optimization using high-order regularized models
    Birgin, E. G.
    Gardenghi, J. L.
    Martinez, J. M.
    Santos, S. A.
    Toint, Ph. L.
    [J]. MATHEMATICAL PROGRAMMING, 2017, 163 (1-2) : 359 - 368
  • [4] Lower bounds for finding stationary points II: first-order methods
    Carmon, Yair
    Duchi, John C.
    Hinder, Oliver
    Sidford, Aaron
    [J]. MATHEMATICAL PROGRAMMING, 2021, 185 (1-2) : 315 - 355
  • [5] Carmon Y, 2017, PR MACH LEARN RES, V70
  • [6] ACCELERATED METHODS FOR NONCONVEX OPTIMIZATION
    Carmon, Yair
    Duchi, John C.
    Hinder, Oliver
    Sidford, Aaron
    [J]. SIAM JOURNAL ON OPTIMIZATION, 2018, 28 (02) : 1751 - 1772
  • [7] Worst-case evaluation complexity of non-monotone gradient-related algorithms for unconstrained optimization
    Cartis, C.
    Sampaio, Ph R.
    Toint, Ph L.
    [J]. OPTIMIZATION, 2015, 64 (05) : 1349 - 1361
  • [8] ON THE COMPLEXITY OF STEEPEST DESCENT, NEWTON'S AND REGULARIZED NEWTON'S METHODS FOR NONCONVEX UNCONSTRAINED OPTIMIZATION PROBLEMS
    Cartis, C.
    Gould, N. I. M.
    Toint, Ph. L.
    [J]. SIAM JOURNAL ON OPTIMIZATION, 2010, 20 (06) : 2833 - 2852
  • [9] CARTIS C., 2011, NAXYS172011 U NAM DE
  • [10] Cartis C., 2019, P INT C MATH ICM 201, V3, P3697