Automatic Generation of Dynamic Inference Architecture for Deep Neural Networks

The computational cost of deep neural network(DNN) model can be reduced dramatically by applying different architectures based on the difficulties of each sample, which is named dynamic inference tech. Manually designed dynamic inference framework is hard to be optimal for the dependency on human experience, which is also time-consuming and labor-intensive. In this paper, we provide an auto-designed AB-Net based on the popular dynamic framework BranchyNet, which is inspired by neural architecture search (NAS). To further accelerate the search procedure, we also develop several specific techniques. Firstly, the search space is optimized by the pre-selection of candidate architectures. Then, a neighborhood greedy search algorithm is developed to efficiently find the optimal architecture in the improved search space. Moreover, our scheme can be extended to the multiple-branch cases to further enhance the performance of the AB-Net. We apply the AB-Net on multiple mainstream models and evaluate them on datasets CIFAR10/100. Compared to the handcrafted BranchyNet, the proposed AB-Net is able to achieve 1.57 x computational cost reduction at least even with slight accuracy improvement on CIFAR100. Moreover, the AB-Net also significantly outperforms the S2DNAS on accuracy with similar cost reduction, which is the state-of-the-art automatic dynamic inference architecture.