Collaborative Diffusion for Multi-Modal Face Generation and Editing

被引:40
作者
Huang, Ziqi [1 ]
Chan, Kelvin C. K. [1 ]
Jiang, Yuming [1 ]
Liu, Ziwei [1 ]
机构
[1] Nanyang Technol Univ, S Lab, Singapore, Singapore
来源
2023 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, CVPR | 2023年
关键词
D O I
10.1109/CVPR52729.2023.00589
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Diffusion models arise as a powerful generative tool recently. Despite the great progress, existing diffusion models mainly focus on uni-modal control, i.e., the diffusion process is driven by only one modality of condition. To further unleash the users' creativity, it is desirable for the model to be controllable by multiple modalities simultaneously, e.g. generating and editing faces by describing the age (text-driven) while drawing the face shape (mask-driven). In this work, we present Collaborative Diffusion, where pre-trained uni-modal diffusion models collaborate to achieve multi-modal face generation and editing without re-training. Our key insight is that diffusion models driven by different modalities are inherently complementary regarding the latent denoising steps, where bilateral connections can be established upon. Specifically, we propose dynamic diffuser, a meta-network that adaptively hallucinates multimodal denoising steps by predicting the spatial-temporal influence functions for each pre-trained uni-modal model. Collaborative Diffusion not only collaborates generation capabilities from uni-modal diffusion models, but also integrates multiple uni-modal manipulations to perform multimodal editing. Extensive qualitative and quantitative experiments demonstrate the superiority of our framework in both image quality and condition consistency.
引用
收藏
页码:6080 / 6090
页数:11
相关论文
共 71 条
[1]  
Amit Tomer, 2021, ARXIV211200390
[2]  
[Anonymous], 2017, NeurIPS
[3]  
[Anonymous], 2021, ICML
[4]  
[Anonymous], 2016, ADV NEURAL INFORM PR
[5]  
[Anonymous], 2016, ICML
[6]  
[Anonymous], 2019, CVPR, DOI DOI 10.1109/CVPR.2019.00514
[7]  
[Anonymous], 2019, CVPR, DOI DOI 10.1109/CVPR.2019.00244
[8]  
Austin J., 2021, NEURIPS
[9]  
Avrahami Omri, 2022, CVPR
[10]  
Baranchuk D., 2022, ICLR, P1