A Vibrating Mechanism to Prevent Neural Networks from Overfitting

In this paper, we propose a Vibrating Mechanism, which can be understood as a continuous version of Dropout and can achieve a certain L1 regularization effect at the same time. In Dropout, the parameters are discarded with a probability obeying Bernoulli distribution. While in the proposed Vibrating Mechanism, the parameters are sampled from a truncated Gaussian distribution or uniform distribution. The mean of the distribution is the result trained by the previous iteration, and the standard deviation of the distribution is the absolute value of the previous iteration training result multiplied by a proportional coefficient. Besides, the performance improvement is theoretically analyzed from the perspective of hyperplane segmentation. The effectiveness of the proposed Vibrating Mechanism is demonstrated by applying it for a text classification task. We choose AG dataset for test. The result shows that the Vibrating Mechanism can achieve better classification accuracy without using L1 regularization or Dropout, which verifies the performance improvement of the Vibrating Mechanism.