Compact Deep Neural Networks with l1,1 and l1,2 Regularization

Deep neural networks have demonstrated its superiority in many fields. Its excellent performance relys on quite a lot of parameters used in the network, resulting in a series of problems, including memory and computation requirement and overfitting, which seriously impede the application of deep neural networks in many assignments in practice. A considerable number of model compression methods have been proposed in deep neural networks to reduce the number of parameters used in networks, among which there is one kind of methods persuing sparsity in deep neural networks. In this paper, we propose to combine l(1,1) and l(1,2) norm together as the regularization term to regularize the objective function of the network. We introduce group and l(1,1) can zero out weights in both intergroup and intra-group level. l(1,2) regularizer can obtain intragroup level sparsity and cause even weights among groups. We adopt proximal gradient descent to solve the objective function regularized by our combined regularization. Experimental results demonstrate the effectiveness of the proposed regularizer when comparing it with other baseline regularizers.