Compressing Deep Neural Network: A Black-Box System Identification Approach

This work proposes a new approach to deep neural network (DNN) compression. We employ black-box function approximation techniques from signal processing to compress. DNN, in general, can approximate non-smooth and piecewise smooth functions. With only this assumption, we model the function that the DNN has learnt as a piecewise linear function. This is a standard function approximation approach. We compared our approach with two state-of-the-art techniques - spatial singular value decomposition and channel pruning with weight reconstruction; and one of state-of-practice tool - OpenVINO. Two well known 1D DNN models for time series classification - ResNet and InceptionTime were compressed. Results show that our model yields better compression at comparable losses in accuracy on majority of the datasets.