Revisit last-iterate convergence of mSGD under milder requirement on step size

Understanding convergence of stochastic gradient descent (SGD) based optimization algorithms can help deal with enormous machine learning problems. To ensure last-iterate convergence of SGD and momentum-based SGD (mSGD), the existing studies usually constrain the step size epsilon(n) to decay as Sigma(+infinity.)(n=1)epsilon(2)(n) < +infinity, which however is rather conservative and may lead to slow convergence in the early stage of the iteration. In this paper, we relax this requirement by studying an alternate s Ptep size for the mSGD. First, we relax the requirement of the decay on step size to Sigma(+infinity.)(n=1)epsilon(2)(n) (+eta 0)< +infinity (0 <= eta(0) < 1/2). This implies that a larger step size, such as epsilon(n) = 1/root n can be utilized for accelerating the mSGD in the early stage. Under this new step size and some common conditions, we prove that the gradient norm of mSGD for a class of non-convex loss functions asymptotically decays to zero. In addition, we show that this step size can indeed help make the iterates of mSGD converge into a neighborhood of the stationary points quicker in the early stage. Finally, we establish the convergence of mSGD under a constant step size epsilon(n) = epsilon > 0 by removing a common requirement in the literature on strong convexity of the loss functions. Some experiments are given to illustrate the developed results.