A Variational U-Net for Conditional Appearance and Shape Generation

被引:290
作者
Esser, Patrick [1 ]
Sutter, Ekaterina [1 ]
Ommer, Bjoern [1 ]
机构
[1] Heidelberg Univ, IWR, Heidelberg Collaboratory Image Proc, Heidelberg, Germany
来源
2018 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR) | 2018年
关键词
D O I
10.1109/CVPR.2018.00923
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Deep generative models have demonstrated great performance in image synthesis. However, results deteriorate in case of spatial deformations, since they generate images of objects directly, rather than modeling the intricate interplay of their inherent shape and appearance. We present a conditional U-Net [30] for shape-guided image generation, conditioned on the output of a variational autoencoder for appearance. The approach is trained end-to-end on images, without requiring samples of the same object with varying pose or appearance. Experiments show that the model enables conditional image generation and transfer. Therefore, either shape or appearance can be retained from a query image, while freely altering the other. Moreover, appearance can be sampled due to its stochastic latent representation, while preserving shape. In quantitative and qualitative experiments on COCO [20], DeepFashion [21, 23], shoes [43], Market-1501 [47] and handbags [49] the approach demonstrates significant improvements over the state-of-the-art.
引用
收藏
页码:8857 / 8866
页数:10
相关论文
共 50 条
[21]   Lung Parenchyma Segmentation Based on U-Net Fused With Shape Stream [J].
Zhu, Lun ;
Cai, Yinghui ;
Liao, Jiahao ;
Wu, Fan .
IEEE ACCESS, 2024, 12 :29238-29251
[22]   MSR U-Net: An Improved U-Net Model for Retinal Blood Vessel Segmentation [J].
Kande, Giri Babu ;
Ravi, Logesh ;
Kande, Nitya ;
Nalluri, Madhusudana Rao ;
Kotb, Hossam ;
Aboras, Kareem M. ;
Yousef, Amr ;
Ghadi, Yazeed Yasin ;
Sasikumar, A. .
IEEE ACCESS, 2024, 12 :534-551
[23]   Sharp dense U-Net: an enhanced dense U-Net architecture for nucleus segmentation [J].
Senapati, Pradip ;
Basu, Anusua ;
Deb, Mainak ;
Dhal, Krishna Gopal .
INTERNATIONAL JOURNAL OF MACHINE LEARNING AND CYBERNETICS, 2024, 15 (06) :2079-2094
[24]   Semi-Dense U-Net: A Novel U-Net Architecture for Face Detection [J].
Pai, Ganesh ;
Kumari, M. Sharmila .
INTERNATIONAL JOURNAL OF ADVANCED COMPUTER SCIENCE AND APPLICATIONS, 2023, 14 (06) :406-414
[25]   MSR U-Net: An Improved U-Net Model for Retinal Blood Vessel Segmentation [J].
Kande, Giri Babu ;
Ravi, Logesh ;
Kande, Nitya ;
Nalluri, Madhusudana Rao ;
Kotb, Hossam ;
Aboras, Kareem M. ;
Yousef, Amr ;
Ghadi, Yazeed Yasin ;
Sasikumar, A. .
IEEE Access, 2024, 12 :534-551
[26]   Attention U-Net for Binary Mask Generation in Medical Microwave Imaging [J].
Yang, Yankai ;
Xue, Fei ;
Guo, Lei ;
Abbosh, Amin .
2024 IEEE INTERNATIONAL SYMPOSIUM ON ANTENNAS AND PROPAGATION AND INC/USNCURSI RADIO SCIENCE MEETING, AP-S/INC-USNC-URSI 2024, 2024, :2761-2762
[27]   Graph isomorphism U-Net [J].
Amouzad, Alireza ;
Dehghanian, Zahra ;
Saravani, Saeed ;
Amirmazlaghani, Maryam ;
Roshanfekr, Behnam .
EXPERT SYSTEMS WITH APPLICATIONS, 2024, 236
[28]   Texture-Guided U-Net for OCT-to-OCTA Generation [J].
Zhang, Ziyue ;
Ji, Zexuan ;
Chen, Qiang ;
Yuan, Songtao ;
Fan, Wen .
PATTERN RECOGNITION AND COMPUTER VISION, PT IV, 2021, 13022 :42-52
[29]   Generation of orbital angular momentum hologram using a modified U-net [J].
Zhang, Zhi-Gang ;
Han, Fei-Fei ;
Wang, Le ;
Zhao, Sheng-Mei .
CHINESE PHYSICS B, 2024, 33 (03)
[30]   BUILDING FOOTPRINT GENERATION BY INTEGRATING U-NET WITH DEEPENED SPACE MODULE [J].
Chen, Jun ;
Jiang, Yuxuan ;
Luo, Linbo ;
Gu, Yue ;
Wu, Kangle .
2021 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 2021, :3847-3851