An investigation of stochastic trust-region based algorithms for finite-sum minimization

This work elaborates on the TRust-region-ish (TRish) algorithm, a stochastic optimization method for finite-sum minimization problems proposed by Curtis et al. in [F.E. Curtis, K. Scheinberg, and R. Shi, A stochastic trust region algorithm based on careful step normalization, INFORMS. J. Optim. 1(3) (2019), pp. 200-220; F.E. Curtis and R. Shi, A fully stochastic second-order trust region method, Optim. Methods Softw. 37(3) (2022), pp. 844-877]. A theoretical analysis that complements the results in the literature is presented, and the issue of tuning the involved hyper-parameters is investigated. Our study also focuses on a practical version of the method, which computes the stochastic gradient by means of the inner product test and the orthogonality test proposed by Bollapragada et al. in [R. Bollapragada, R. Byrd, and J. Nocedal, Adaptive sampling strategies for stochastic optimization, SIAM. J. Optim. 28(4) (2018), pp. 3312-3343]. It is shown experimentally that this implementation improves the performance of TRish and reduces its sensitivity to the choice of the hyper-parameters.