Ranked Dropout for Handwritten Digit Recognition

Overfitting is a common problem in training of neural network with small training sets, which leads to worse performance on the new samples. Dropout has been proved to be an effective method to avoid overfitting, which prevents co-adaptation of features detectors by randomly discarding nodes from hidden layers of network. Inspired by dropout, we proposed a ranked dropout method to remove randomness of standard dropout mask, which discards a part of active nodes and forces the inactive nodes to learn more features to improve generalization ability. We apply the proposed ranked dropout to a stacked autoencoder network and compare it with standard dropout, gaussian dropout, uniform dropout and DropConnect on MNIST dataset. Experimental results of handwritten digit recognition demonstrate that the ranked strategy leads to better classification performance and the proposed ranked dropout can effectively reduce interference of overfitting and improve model's generalization ability.