Adaptive lightweight network construction method for Self-Knowledge Distillation

Self-Knowledge Distillation (self-KD) has become a promising method for neural network compression due to its advances in computational efficiency. Nevertheless, its applicability is constrained by the inherent inflexibility of the network architecture and the absence of quantitative metrics to evaluate the distillability of the architecture. To address these problems, a two-stage adaptive dynamic distillation network framework (ADDN) is proposed to adapt the architecture based on the distillability of the current architecture, containing a hypernetwork topology construction stage and a subnetwork training stage. To evaluate the distillability of current architectures without necessitating extensive training, we propose a set of low-cost distillability metrics that evaluate architectures from the perspective of architectural similarity and clustering ability. Furthermore, to simplify the hypernetwork structure and reduce the complexity of the construction process, a hierarchical filtering module is introduced to dynamically refine and remove candidate operations within the architecture incrementally, contingent upon the distillability of the current architecture. To validate the effectiveness of our approach, we conduct extensive experiments on various image classification datasets and compare with current works. Experimental results demonstrate that the self-knowledge distillation network architecture obtained by our proposed methodology simultaneously attains superior distillability and efficiency while significantly curtailing construction expenses.