An Empirical Study on Position of the Batch Normalization Layer in Convolutional Neural Networks

In this paper, we have studied how training of the convolutional neural networks (CNNs) can be affected by changing the position of the batch normalization (BN) layer. Three different convolutional neural networks have been chosen for our experiments. These networks are AlexNet, VGG-16, and ResNet-20. We show that the speed-up provided by the BN algorithm can be further improved by using the BN in positions other than the one suggested by its original paper. Also, we discuss how the BN layer in a certain position can aid the training of one network but not the other. Three different positions for the BN layer have been studied in this research, these positions are: BN layer between the convolution layer and the non-linear activation function, BN layer after the non-linear activation function and finally, the BN layer before each of the convolutional layers.