An adaptive Drop method for deep neural networks regularization: Estimation of DropConnect hyperparameter using generalization gap

Deep neural networks (DNNs) have solved numerous challenging reels problems. However, successful DNNs often require a large number of parameters, which may produce some undesirable phenomena, notably overfitting. DropConnect, which is a generalization of Dropout, is one of the successful stochastic regularizers that prevent overfitting in deep neural networks. Indeed, it allows dropping some parameters according to a fixed probability, generating at the end dynamic sparse DNNs. The study of the DropConnect hyperparameter is still not estimated and needs a theoretical understanding. In this context, we propose an estimation of the DropConnect hyperparameter using the gap generalization and the Rademacher complexity. This estimation gives rise to a new DropConnect technique named Adaptive DropConnect (A-DropConnect), in which the studied hyperparameter is updated using the data-dependent during training. Efficiency of A-DropConnect is demonstrated by several experiments. Numerical results demonstrate that the proposed method yields significant improvement in classification performance compared to the state of the art. (C) 2022 Elsevier B.V. All rights reserved.