A stochastic recursive gradient algorithm integrating momentum and the powerball function with adaptive step sizes

Momentum techniques and the Powerball function have been proven effective in stochastic optimization algorithms, widely utilized in large-scale optimization scenarios. Nonetheless, the integration of these methodologies into stochastic optimization algorithms and the determination of their initial learning rates persist as unresolved and pivotal issues. In this study, we integrate momentum techniques and the Powerball function into the SARAH (StochAstic Recursive grAdient algoritHm), culminating in the inception of a novel variance-reduced gradient descent algorithm named PM-SARAH. Moreover, a pair of adaptive step size variants are respectively integrated into the outer and inner loops of PM-SARAH, giving rise to PM-SARAH-AS and PM-SARAH-RAS. Ultimately, through comparative experimentation with state-of-the-art optimization algorithms on standard machine learning tasks and certain non-convex scenarios, the empirical results underscore the superior performance of the algorithms elucidated in this paper.