Prototypical Bidirectional Adaptation and Learning for Cross-Domain Semantic Segmentation

被引:10
作者
Ren, Qinghua [1 ]
Mao, Qirong [1 ,2 ]
Lu, Shijian [3 ]
机构
[1] Jiangsu Univ, Sch Comp Sci & Commun Engn, Zhenjiang 212013, Jiangsu, Peoples R China
[2] Jiangsu Engn Res Ctr Big Data Ubiquitous Percept &, Zhenjiang 212013, Jiangsu, Peoples R China
[3] Nanyang Technol Univ, Sch Comp Sci & Engn, Singapore 639798, Singapore
基金
中国国家自然科学基金;
关键词
Semantic segmentation; domain adaptation; bidirectional adaptation; prototypical learning; REPRESENTATION; CONTRAST;
D O I
10.1109/TMM.2023.3266892
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
Cross-domain semantic segmentation, which aims to address the distribution shift while adapting from a labeled source domain to an unlabeled target domain, has achieved great progress in recent years. However, most existing work adopts a source-to-target adaptation path, which often suffers from clear class mismatching or class imbalance issues. We design PBAL, a prototypical bidirectional adaptation and learning technique that introduces bidirectional prototype learning and prototypical self-training for optimal inter-domain alignment and adaptation. We perform bidirectional alignments in a complementary and cooperative manner which balances both dominant and tail categories as well as easy and hard samples effectively. In addition, We derive prototypes efficiently from a source-trained classifier, which enables class-aware adaptation as well as synchronous prototype updating and network optimization. Further, we re-examine self-training and introduce prototypical contrast above it which greatly improves inter-domain alignment by promoting better intra-class compactness and inter-class separability in the feature space. Extensive experiments over two widely studied benchmarks show that the proposed PBAL achieves superior domain adaptation performance as compared with the state-of-the-art.
引用
收藏
页码:501 / 513
页数:13
相关论文
共 61 条
[1]   Self-supervised Augmentation Consistency for Adapting Semantic Segmentation [J].
Araslanov, Nikita ;
Roth, Stefan .
2021 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, CVPR 2021, 2021, :15379-15389
[2]   All about Structure: Adapting Structural Information across Domains for Boosting Semantic Segmentation [J].
Chang, Wei-Lun ;
Wang, Hui-Po ;
Peng, Wen-Hsiao ;
Chiu, Wei-Chen .
2019 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2019), 2019, :1900-1909
[3]   Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation [J].
Chen, Liang-Chieh ;
Zhu, Yukun ;
Papandreou, George ;
Schroff, Florian ;
Adam, Hartwig .
COMPUTER VISION - ECCV 2018, PT VII, 2018, 11211 :833-851
[4]   DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs [J].
Chen, Liang-Chieh ;
Papandreou, George ;
Kokkinos, Iasonas ;
Murphy, Kevin ;
Yuille, Alan L. .
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 2018, 40 (04) :834-848
[5]  
Chen T., 2020, Advances in Neural Information Processing Systems, V33, P22243, DOI 10.48550/arXiv.2006.10029
[6]   Enhanced Feature Alignment for Unsupervised Domain Adaptation of Semantic Segmentation [J].
Chen, Tao ;
Wang, Shui-Hua ;
Wang, Qiong ;
Zhang, Zheng ;
Xie, Guo-Sen ;
Tang, Zhenmin .
IEEE TRANSACTIONS ON MULTIMEDIA, 2022, 24 :1042-1054
[7]   Exploring Simple Siamese Representation Learning [J].
Chen, Xinlei ;
He, Kaiming .
2021 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, CVPR 2021, 2021, :15745-15753
[8]   CrDoCo: Pixel-level Domain Transfer with Cross-Domain Consistency [J].
Chen, Yun-Chun ;
Lin, Yen-Yu ;
Yang, Ming-Hsuan ;
Huang, Jia-Bin .
2019 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2019), 2019, :1791-1800
[9]   Self-Ensembling with GAN-based Data Augmentation for Domain Adaptation in Semantic Segmentation [J].
Choi, Jaehoon ;
Kim, Taekyung ;
Kim, Changick .
2019 IEEE/CVF INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV 2019), 2019, :6829-6839
[10]   The Cityscapes Dataset for Semantic Urban Scene Understanding [J].
Cordts, Marius ;
Omran, Mohamed ;
Ramos, Sebastian ;
Rehfeld, Timo ;
Enzweiler, Markus ;
Benenson, Rodrigo ;
Franke, Uwe ;
Roth, Stefan ;
Schiele, Bernt .
2016 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 2016, :3213-3223