ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Experts

被引:46
作者
Feng, Zhida [1 ,2 ]
Zhang, Zhenyu [1 ]
Yu, Xintong [1 ]
Fang, Yewei [1 ]
Li, Lanxin [1 ]
Chen, Xuyi [1 ]
Lu, Yuxiang [1 ]
Liu, Jiaxiang [1 ]
Yin, Weichong [1 ]
Feng, Shikun [1 ]
Sun, Yu [1 ]
Chen, Li [2 ]
Tian, Hao [1 ]
Wu, Hua [1 ]
Wang, Haifeng [1 ]
机构
[1] Baidu Inc, Beijing, Peoples R China
[2] Wuhan Univ Sci & Technol, Sch Comp Sci & Technol, Wuhan, Peoples R China
来源
2023 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR) | 2023年
基金
中国国家自然科学基金;
关键词
D O I
10.1109/CVPR52729.2023.00977
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Recent progress in diffusion models has revolutionized the popular technology of text-to-image generation. While existing approaches could produce photorealistic high-resolution images with text conditions, there are still several open problems to be solved, which limits the further improvement of image fidelity and text relevancy. In this paper, we propose ERNIE-ViLG 2.0, a large-scale Chinese text-to-image diffusion model, to progressively upgrade the quality of generated images by: (1) incorporating fine-grained textual and visual knowledge of key elements in the scene, and (2) utilizing different denoising experts at different denoising stages. With the proposed mechanisms, ERNIE-ViLG 2.01 not only achieves a new state-of-the-art on MS-COCO with zero-shot FID-30k score of 6.75, but also significantly outperforms recent models in terms of image fidelity and image-text alignment, with side-by-side human evaluation on the bilingual prompt set ViLG-300.
引用
收藏
页码:10135 / 10145
页数:11
相关论文
共 47 条
[1]   Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering [J].
Anderson, Peter ;
He, Xiaodong ;
Buehler, Chris ;
Teney, Damien ;
Johnson, Mark ;
Gould, Stephen ;
Zhang, Lei .
2018 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 2018, :6077-6086
[2]  
[Anonymous], 2021, PMLR
[3]  
Ding Ming, 2022, ABS220414217 CORR
[4]  
Ding Ming, 2021, Advances in Neural Information Processing Systems, V34
[5]  
Dong Li, 2022, ABS220810442 CORR
[6]  
Fedus William, 2022, ABS220901667 CORR
[7]  
Fedus William, 2021, CoRR, abs/2101.03961.
[8]  
Gafni Oran, 2022, ABS220313131 CORR
[9]  
Goodfellow IJ, 2014, ADV NEUR IN, V27, P2672
[10]  
Gururangan S, 2022, NAACL 2022: THE 2022 CONFERENCE OF THE NORTH AMERICAN CHAPTER OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS: HUMAN LANGUAGE TECHNOLOGIES, P5557