Knowledge Distillation for Reduced Footprint Semantic Segmentation with the U-Net

Model compression techniques such as knowledge distillation, pruning, and quantization are well documented in the computer vision literature for image classification and localization tasks. On the other hand, and possibly due to its higher computational cost, semantic segmentation at small footprint devices has significantly less references. We present a case study of knowledge distillation for semantic segmentation, from relatively large pre-trained networks (a ResNet-50 and a U-Net with ResNet-50 backbone) to a compact flavor of the U-Net. The distillation is performed at several levels of the student U-Net, and then compared.