Cross-supervision-based equilibrated fusion mechanism of local and global attention for semantic segmentation

In recent years, weakly supervised semantic segmentation has become one of the hot research directions, but the problems of object location accuracy and small activation areas are still a challenge. In this paper, we propose a network with an equilibrated fusion mechanism of local and global attention based on cross-supervision(CSEFM). To accomplish multitask learning, we design the class activation branch and decoding prediction branch, which share the same backbone, to generate high-quality pseudo labels and obtain semantic segmentation results. Specifically, the network is first trained to update the backbone weights using images with strong labels. Images with weak labels are then used to help obtain a reliable class activation mapping (CAM) at the class activation branch, and the dense conditional random fields (DenseCRFs) are used to generate high-quality pseudo labels. Finally, the strong label images and the weak label images that have obtained pseudo labels are fed to the network for retraining, and the result of segmentation is predicted by the decoding prediction branch. Given the training method, the dataset is divided into several groups as a few images with strong labels and several images with weak labels and fed to the network successively for cross-supervised learning. Our proposed network is trained and validated on the PASCAL VOC 2012 dataset, and the results show that the mean Intersection over Union (mIoU) on the validation set is 65.6%. Compared with other mainstream methods, better segmentation is achieved and the performance gap between image-level and pixel-level semantic segmentation is reduced when using our approach.