Streaming First-Order Optimization Methods With Momentum for Dominant Eigenspace Estimation

In this article, we consider the dominant eigenspace recovery for PCA in the streaming scenario, and design several new algorithms for this problem by incorporating various momentum schemes into two classical gradient based streaming eigensolvers, which emerges as a frequently-used trick in improving gradient type methods. Theoretically, we establish the convergence guarantees of the proposed first-order momentum methods, and prove their decreasing variance under favorable conditions. Extensive numerical experiments are conducted to corroborate the result that momentum does yield variance reduction, and also highlight the advantages of lowering the stepsize sensitivity. In addition, our methods with appropriately chosen momentum yield a dramatically enhanced performance in terms of both the convergence speed and accuracy.