Pull and concentrate: improving unsupervised semantic segmentation adaptation with cross- and intra-domain consistencies

Unsupervised domain adaptation (UDA) is an important solution for the cross-domain problem in semantic segmentation. Existing segmentation UDA methods mainly consider the domain shift as the major challenge. This paper, from a novel viewpoint, disentangles the cross-domain problem into two negative factors beyond the domain shift. Specifically, we find that apart from the domain shift factor, the dispersed within-class distribution on the target domain is another factor that compromises cross-domain segmentation. This paper finds that the neglected target domain distribution dispersion is a challenge as crucial as the domain shift. In response to the joint of these two negative factors, we propose a "Pull-and-Concentrate" (PuCo) method comprised of two consistencies: (1) A cross-domain consistency "pulls" the source and target domain distribution (of the same class) close to each other based on a novel statistical style transfer. (2) An intra-domain consistency "concentrates" the within-class distribution on the target domain in a new unsupervised teacher-student method. Both consistencies have the advantage of being robust (or insulated) from pseudo-label noises. This advantage allows PuCo to bring consistent improvement over a battery of pseudo-label-based UDA methods. For example, on GTA5 to Cityscapes and SYNTHIA to Cityscapes, PuCo achieves 60.3% and 57.2% mean IoU, respectively. Code is available at https://github.com/Jarvis73/PuCo.