A Variational U-Net for Conditional Appearance and Shape Generation

被引:290
作者
Esser, Patrick [1 ]
Sutter, Ekaterina [1 ]
Ommer, Bjoern [1 ]
机构
[1] Heidelberg Univ, IWR, Heidelberg Collaboratory Image Proc, Heidelberg, Germany
来源
2018 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR) | 2018年
关键词
D O I
10.1109/CVPR.2018.00923
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Deep generative models have demonstrated great performance in image synthesis. However, results deteriorate in case of spatial deformations, since they generate images of objects directly, rather than modeling the intricate interplay of their inherent shape and appearance. We present a conditional U-Net [30] for shape-guided image generation, conditioned on the output of a variational autoencoder for appearance. The approach is trained end-to-end on images, without requiring samples of the same object with varying pose or appearance. Experiments show that the model enables conditional image generation and transfer. Therefore, either shape or appearance can be retained from a query image, while freely altering the other. Moreover, appearance can be sampled due to its stochastic latent representation, while preserving shape. In quantitative and qualitative experiments on COCO [20], DeepFashion [21, 23], shoes [43], Market-1501 [47] and handbags [49] the approach demonstrates significant improvements over the state-of-the-art.
引用
收藏
页码:8857 / 8866
页数:10
相关论文
共 50 条
[31]   Generation of orbital angular momentum hologram using a modified U-net [J].
郑志刚 ;
韩菲菲 ;
王乐 ;
赵生妹 .
ChinesePhysicsB, 2024, 33 (03) :460-466
[32]   MVP U-Net: Multi-View Pointwise U-Net for Brain Tumor Segmentation [J].
Zhao, Changchen ;
Zhao, Zhiming ;
Zeng, Qingrun ;
Feng, Yuanjing .
BRAINLESION: GLIOMA, MULTIPLE SCLEROSIS, STROKE AND TRAUMATIC BRAIN INJURIES (BRAINLES 2020), PT II, 2021, 12659 :93-103
[33]   Pixel U-Net: an improved version of U-Net for binary segmentation of wind turbine blades [J].
Rizvi, Syed Zeeshan ;
Jamil, Mohsin ;
Huang, Weimin .
SIGNAL IMAGE AND VIDEO PROCESSING, 2024, 18 (8-9) :6299-6307
[34]   Modifying U-Net for small dataset - a simplified U-Net version for Liver Parenchyma segmentation [J].
Prasad, Pravda Jith Ray ;
Elle, Ole Jakob ;
Lindseth, Frank ;
Albregtsen, Fritz ;
Kumar, Rahul Prasanna .
MEDICAL IMAGING 2021: COMPUTER-AIDED DIAGNOSIS, 2021, 11597
[35]   GT U-Net: A U-Net Like Group Transformer Network for Tooth Root Segmentation [J].
Li, Yunxiang ;
Wang, Shuai ;
Wang, Jun ;
Zeng, Guodong ;
Liu, Wenjun ;
Zhang, Qianni ;
Jin, Qun ;
Wang, Yaqi .
MACHINE LEARNING IN MEDICAL IMAGING, MLMI 2021, 2021, 12966 :386-395
[36]   E-Res U-Net: An improved U-Net model for segmentation of muscle images [J].
Zhou, Junsheng ;
Lu, Yiwen ;
Tao, Siyi ;
Cheng, Xuan ;
Huang, Chenxi .
EXPERT SYSTEMS WITH APPLICATIONS, 2021, 185
[37]   Underwater U-Net: Deep Learning with U-Net for Visual Underwater Moving Object detection [J].
Bajpai, Vatsalya ;
Sharma, Akhilesh ;
Subudhi, Badri Narayan ;
Veerakumar, T. ;
Jakhetiya, Vinit .
OCEANS 2021: SAN DIEGO - PORTO, 2021,
[38]   E-Res U-Net: An improved U-Net model for segmentation of muscle images [J].
Zhou, Junsheng ;
Lu, Yiwen ;
Tao, Siyi ;
Cheng, Xuan ;
Huang, Chenxi .
Expert Systems with Applications, 2021, 185
[39]   A shape-supervised feature fusion U-Net for tubular structure segmentation [J].
Yue, Jinghua ;
Jin, Shuo ;
Wang, Siyuan ;
Zeng, Jianping ;
Shan, Siqiao ;
Liu, Bo ;
Jiang, Nan ;
Zhou, Fugen .
COMPUTERS & ELECTRICAL ENGINEERING, 2024, 119
[40]   MA-Res U-Net: Design of Soybean Navigation System with Improved U-Net Model [J].
Liu, Qianshuo ;
Zhao, Jun .
PHYTON-INTERNATIONAL JOURNAL OF EXPERIMENTAL BOTANY, 2024, 93 (10) :2663-2681