Node pruning based on Entropy of Weights and Node Activity for Small-footprint Acoustic Model based on Deep Neural Networks

This paper describes a node-pruning method for an acoustic model based on deep neural networks (DNNs). Node pruning is a promising method to reduce the memory usage and computational cost of DNNs. A score function is defined to measure the importance of each node, and less important nodes are pruned. The entropy of the activity of each node has been used as a score function to find nodes with outputs that do not change at all. We introduce entropy of weights of each node to consider the number of weights and their patterns of each node. Because the number of weights and the patterns differ at each layer, the importance of the node should also be measured using the related weights of the target node. We then propose a score function that integrates the entropy of weights and node activity, which will prune less important nodes more efficiently. Experimental results showed that the proposed pruning method successfully reduced the number of parameters by about 6% without any accuracy loss compared with a score function based only on the entropy of node activity.