Lightweight image dehazing networks based on soft knowledge distillation

A two-stage knowledge transfer framework for distilling efficient dehazing networks is proposed in this paper. Recently, lightweight dehazing studies based on knowledge distillation have shown great promise and potential. However, existing approaches have only focused on exploiting knowledge extracted from clean images (hard knowledge) while neglecting the concise knowledge encoded from hazy images (soft knowledge). Additionally, recent methods have solely emphasized process-oriented learning rather than response-oriented learning. Motivated by these observations, the proposed framework is targeted toward aptly exploiting soft knowledge and response-oriented learning to produce improved dehazing models. A general encoder-decoder dehazing structure is utilized as the teacher network as well as a basis for constructing the student model with drastic complexity reduction using a channel multiplier. A transmission-aware loss is adopted that leverages the transmission information to enhance the network's generalization ability across different haze densities. The derived network, called soft knowledge-based distilled dehazing network (SDDN), achieves a significant reduction in complexity while maintaining satisfactory performance or even showing better generalization capability in certain cases. Experiments on various benchmark datasets have demonstrated that SDDN can be compared competitively with prevailing dehazing approaches. Moreover, SDDN shows a promising applicability to intelligent driving systems. When combined with YOLOv4, SDDN can improve the detection performance under hazy weather by 9.1% with only a negligible increase in the number of parameters (0.87%). The code of this work is publicly available at https://github.com/tranleanh/sddn.