Accelerated affine-invariant convergence rates of the Frank-Wolfe algorithm with open-loop step-sizes

被引:2
作者
Wirth, Elias [1 ]
Pena, Javier [2 ]
Pokutta, Sebastian [1 ]
机构
[1] Berlin Inst Technol, Inst Math, Str 17 Juni 135, Berlin, Germany
[2] Carnegie Mellon Univ, Tepper Sch Business, 5000 Forbes Ave, Pittsburgh, PA 15213 USA
基金
美国安德鲁·梅隆基金会;
关键词
D O I
10.1007/s10107-024-02180-2
中图分类号
TP31 [计算机软件];
学科分类号
081202 ; 0835 ;
摘要
Recent papers have shown that the Frank-Wolfe algorithm (FW) with open-loop step-sizes exhibits rates of convergence faster than the iconic O(t-1)\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {O}(t<^>{-1})$$\end{document} rate. In particular, when the minimizer of a strongly convex function over a polytope lies on the boundary of the polytope, the FW algorithm with open-loop step-sizes eta t=& ell;t+& ell;\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta _t = \frac{\ell }{t+\ell }$$\end{document} for & ell;is an element of N >= 2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ell \in \mathbb {N}_{\ge 2}$$\end{document} has accelerated convergence O(t-2)\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {O}(t<^>{-2})$$\end{document} in contrast to the rate Omega(t-1-& varepsilon;)\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Omega (t<^>{-1-\epsilon })$$\end{document} attainable with more complex line-search or short-step step-sizes. Given the relevance of this scenario in data science problems, research has grown to explore the settings enabling acceleration in open-loop FW. However, despite FW's well-known affine invariance, existing acceleration results for open-loop FW are affine-dependent. This paper remedies this gap in the literature, by merging two recent research trajectories: affine invariance (Pe & ntilde;a in SIAM J. Optim. 33(4):2654-2674, 2023) and open-loop step-sizes (Wirth et al. in Proceedings of the International Conference on Artificial Intelligence and Statistics, 2023). In particular, we extend all known non-affine-invariant convergence rates for FW with open-loop step-sizes to affine-invariant results.
引用
收藏
页数:45
相关论文
共 47 条
[31]  
Levitin E. S., 1966, USSR Computational mathematics and mathematical physics, V6, P1, DOI DOI 10.1016/0041-5553(66)90114-5
[32]   A Momentum-Guided Frank-Wolfe Algorithm [J].
Li, Bingcong ;
Coutino, Mario ;
B. Giannakis, Georgios ;
Leus, Geert .
IEEE TRANSACTIONS ON SIGNAL PROCESSING, 2021, 69 :3597-3611
[33]  
Macdonald Jan, 2022, P MACHINE LEARNING R
[34]   AN ELEMENTARY PROOF OF FORMULA SIGMAINFINITYK=1 1/K2=PI2/6 [J].
MATSUOKA, Y .
AMERICAN MATHEMATICAL MONTHLY, 1961, 68 (05) :485-&
[35]  
Mehta B, 2007, RECSYS 07: PROCEEDINGS OF THE 2007 ACM CONFERENCE ON RECOMMENDER SYSTEMS, P49
[36]   SCALABLE ROBUST MATRIX RECOVERY: FRANK-WOLFE MEETS PROXIMAL METHODS [J].
Mu, Cun ;
Zhang, Yuqian ;
Wright, John ;
Goldfarb, Donald .
SIAM JOURNAL ON SCIENTIFIC COMPUTING, 2016, 38 (05) :A3291-A3317
[37]  
Ravi SN, 2018, Arxiv, DOI arXiv:1803.06453
[38]   Complexity bounds for primal-dual methods minimizing the model of objective function [J].
Nesterov, Yu. .
MATHEMATICAL PROGRAMMING, 2018, 171 (1-2) :311-330
[39]  
Paty FP, 2019, PR MACH LEARN RES, V97
[40]  
Pedregosa F, 2020, Arxiv, DOI arXiv:1806.05123