How to Train Your SSAG in Convolutional Network

SSAG(Stochastic Stratified Average Gradient) is an optimization algorithm that achieves linear convergence with smaller storage and iteration costs. However, in the convolutional network, SSAG has two problems. One is that the training is unstable, and it has high requirements on the learning rate, which increases the training difficulty of the model. The other is that Batch Normalization does not work in the SSAG algorithm, making the SSAG algorithm unusable in many models, limiting the scope of SSAG use. For these two problems, we propose SSAGM(Stochastic Stratified Average Gradient with Momentous) and SSBN(Stochastic Stratified Batch Normalization). SSAGM draws on the SGDM (Stochastic Gradient Descent with Momentum) algorithm, and uses exponentially weighted moving average to estimate the first-order moment of the gradient of each category, which makes the model training more stable and converges faster. SSBN draws on SSAG stratified random sampling technology to change the calculation method of mean and variance in Batch Normalization, so that SSAG algorithm can also perform normalization. The experimental results show that SSAGM is better than SSAG and other algorithms, and SSBN can also play a role similar to Batch Normalization in SSAG algorithm.