Improving Lightweight AdderNet via Distillation From ℓ2 to ℓ1-norm

To achieve efficient inference with a hardware- friendly design, Adder Neural Networks (ANNs) are proposed to replace expensive multiplication operations in Convolutional Neural Networks (CNNs) with cheap additions through utilizing l(1)-norm for similarity measurement instead of cosine distance. However, we observe that there exists an increasing gap between CNNs and ANNs with reducing parameters, which cannot be eliminated by existing algorithms. In this paper, we present a simple yet effective Norm-Guided Distillation (NGD) method for l(1)-norm ANNs to learn superior performance from l(2)-norm ANNs. Although CNNs achieve similar accuracy with l(2)-norm ANNs, the clustering performance based on l(2)-distance can be easily learned by l(1)-norm ANNs compared with cross correlation in CNNs. The features in l(2)-norm ANNs are encouraged to achieve intra-class centralization and inter-class decentralization to amplify this advantage. Furthermore, the roughly estimated gradients in vanilla ANNs are modified to a progressive approx- imation from l(2)-norm to l(1)-norm so that a more accurate optimization can be achieved. Extensive evaluations on several benchmarks demonstrate the effectiveness of NGD on lightweight networks. For example, our method improves ANN by 10.43% with 0.25x GhostNet on CIFAR-100 and 3.1% with 1.0x GhostNet on ImageNet.