FreeNet: An efficient frequency-domain early exiting network for dynamic inference

Early exiting has become an effective method for improving inference efficiency. Most early exiting models place early classifiers in shallow layers, forcing reliance on low-level features and limiting performance. Moreover, these models typically use the spatial representation of the image (RGB image) as input, reducing spatial redundancy only by lowering the resolution. Compared to spatial representation, frequency domain representation provides richer multilevel information, enabling more efficient early exiting mechanisms. To this end, this paper proposes a dynamic frequency-domain early exiting network (FreeNet). Our model employs wavelet transform to decompose images into multiple levels of frequency information, with distinct frequency sub-models dedicated to feature extraction at each level. Each classifier is positioned at the end of its frequency sub-model, ensuring it can leverage high-level features with semantic information. Our model performs inference sequentially from low frequency to high frequency. For easy samples, it uses only a small amount of low-frequency information, while for hard samples, it dynamically supplements high-frequency information to ensure reliable results. Extensive experiments applying our method to various baseline models show that our model achieves abetter balance between performance and computational cost. For instance, on the ImageNet-1k dataset, our method can reduce 58% FLOPs of ResNet50 by only a 0.3% accuracy loss.