Momentum-based variance-reduced stochastic Bregman proximal gradient methods for nonconvex nonsmooth optimization

In machine learning, it is common to encounter a class of nonconvex, nonsmooth composite optimization problems where the objective function may not possess a globally Lipschitz continuous gradient. To address such issues, we investigate stochastic Bregman proximal gradient methods with recursive momentum, and propose SBPG-STORM, whose convergence in expectation is established under bounded variance assumption. Although this assumption is widely used, it is often not satisfied in many practical applications. One possible method to solve this problem is to use variance reduction estimators, which we have integrated with recursive momentum to propose SBPG-SVRRM and SBPG-SVRRM+. Without assuming bounded variance, we prove the convergence in expectation by establishing their descent Lyapunov functions. Furthermore, under the expectation K & Lstrok; property, we demonstrate the global convergence of the iteration points as well as the convergence speed under different & Lstrok;ojasiewicz exponents. The preliminary numerical results indicate that our proposed methods exhibit comparable performance to the state-of-the-art methods.