On Stochastic Gradient Langevin Dynamics with Dependent Data Streams: The Fully Nonconvex Case

被引：30

作者：

Chau, Ngoc Huy ^{[1
]}

Moulines, Eric ^{[2
]}

Rasonyi, Miklos ^{[3
]}

Sabanis, Sotirios ^{[4
]}

Zhang, Ying ^{[5
]}

机构：

[1] Osaka Univ, Ctr Math Modeling & Data Sci, Toyonaka, Osaka 5608531, Japan

[2] Ecole Polytech, Palaiseau, France

[3] Alfred Renyi Inst Math, Budapest, Hungary

[4] Univ Edinburgh, Math, Edinburgh EH9 3JZ, Midlothian, Scotland

[5] Nanyang Technol Univ, Singapore, Singapore

来源：

SIAM JOURNAL ON MATHEMATICS OF DATA SCIENCE | 2021年 / 3卷 / 03期

基金：

英国工程与自然科学研究理事会;

关键词：

stochastic gradient; Langevin dynamics; nonconvex optimization; convergence guarantees; contraction; ORDERS;

D O I：

10.1137/20M1355392

中图分类号：

O29 [应用数学];

学科分类号：

070104 ;

摘要：

We consider the problem of sampling from a target distribution, which is not necessarily log-concave, in the context of empirical risk minimization and stochastic optimization as presented in [M. Raginsky, A. Rakhlin, and M. Telgarsky, Proc. Mach. Learn. Res., 65 (2017), pp. 1674-1703]. Non-asymptotic results are established in the L-1-Wasserstein distance for the behavior of stochastic gradient Langevin dynamics algorithms. We allow gradient estimates based on dependent data streams. Our convergence estimates are sharper and uniform in the number of iterations, in contrast to those in previous studies.

引用

页码：959 / 986

页数：28

共 43 条

[1]

Assran M, 2020, PR MACH LEARN RES, V119

[2] On stochastic gradient Langevin dynamics with dependent data streams in the logconcave case [J].

Barkhagen, M. ;

Chau, N. H. ;

Moulines, E. ;

Rasonyi, M. ;

Sabanis, S. ;

Zhang, Y. .

BERNOULLI, 2021, 27 (01) :1-33

[3] The tamed unadjusted Langevin algorithm [J].

Brosse, Nicolas ;

Durmus, Alain ;

Moulines, Eric ;

Sabanis, Sotirios .

STOCHASTIC PROCESSES AND THEIR APPLICATIONS, 2019, 129 (10) :3638-3663

[4]

Chau H. N., 2020, ARXIV190310328

[5] ON FIXED GAIN RECURSIVE ESTIMATORS WITH DISCONTINUITY IN THE PARAMETERS [J].

Chau, Huy N. ;

Kumar, Chaman ;

Rasonyi, Miklos ;

Sabanis, Sotirios .

ESAIM-PROBABILITY AND STATISTICS, 2019, 23 :217-244

[6]

CHAU N., 2020, ARXIV190513142MATHST

[7]

Cheng X., 2018, Sharp convergence rates for Langevin dynamics in the nonconvex setting

[8]

Cheng Xiang, 2018, Algorithmic learning theory, P186

[9]

Cont R, 2001, QUANT FINANC, V1, P223, DOI [10.1080/713665670, 10.1088/1469-7688/1/2/304]

[10]

Dalalyan A., 2017, PMLR, P678

← 1 2 3 4 5 →