Text-Guided Image Manipulation via Generative Adversarial Network With Referring Image Segmentation-Based Guidance

被引：2
作者：
Watanabe, Yuto [1 ]
Togo, Ren [2 ]
Maeda, Keisuke [2 ]
Ogawa, Takahiro [2 ]
Haseyama, Miki [2 ]
机构：
[1] Hokkaido Univ, Grad Sch Informat Sci & Technol, Sapporo 0600814, Japan
[2] Hokkaido Univ, Fac Informat Sci & Technol, Sapporo 0600814, Japan
来源：
IEEE ACCESS | 2023年 / 11卷
基金：
日本学术振兴会;
关键词：
Image segmentation; Text recognition; Generative adversarial networks; Image color analysis; Visualization; Image reconstruction; Text processing; Text-guided image manipulation; text-to-image synthesis; generative adversarial network; referring image segmentation;
D O I：
10.1109/ACCESS.2023.3269847
中图分类号：
TP [自动化技术、计算机技术];
学科分类号：
0812 ;
摘要：
This study proposes a novel text-guided image manipulation method that introduces referring image segmentation into a generative adversarial network. The proposed text-guided image manipulation method aims to manipulate images containing multiple objects while preserving text-unrelated regions. The proposed method assigns the task of distinguishing between text-related and unrelated regions in an image to segmentation guidance based on referring image segmentation. With this architecture, the adversarial generative network can focus on generating new attributes according to the text description and reconstructing text-unrelated regions. For the challenging input images with multiple objects, the experimental results demonstrate that the proposed method outperforms conventional methods in terms of image manipulation precision.
引用
收藏
页码：42534 / 42545
页数：12
相关论文
共 50 条
[31]   Edge-Guided Generative Adversarial Network for Image Inpainting [J].
Xu, Shunxin ;
Liu, Dong ;
Xiong, Zhiwei .
2017 IEEE VISUAL COMMUNICATIONS AND IMAGE PROCESSING (VCIP), 2017,
[32]   Text-Guided Cross-Position Attention for Segmentation: Case of Medical Image [J].
Lee, Go-Eun ;
Kim, Seon Ho ;
Cho, Jungchan ;
Choi, Sang Tae ;
Choi, Sang-Il .
MEDICAL IMAGE COMPUTING AND COMPUTER ASSISTED INTERVENTION, MICCAI 2023, PT V, 2023, 14224 :537-546
[33]   Hierarchically-fused Generative Adversarial Network for text to realistic image synthesis [J].
Huang, Xin ;
Wang, Mingjie ;
Gong, Minglun .
2019 16TH CONFERENCE ON COMPUTER AND ROBOT VISION (CRV 2019), 2019, :73-80
[34]   Vision-Language Matching for Text-to-Image Synthesis via Generative Adversarial Networks [J].
Cheng, Qingrong ;
Wen, Keyu ;
Gu, Xiaodong .
IEEE TRANSACTIONS ON MULTIMEDIA, 2023, 25 :7062-7075
[35]   An image augmentation approach using two-stage generative adversarial network for nuclei image segmentation [J].
Pandey, Siddharth ;
Singh, Pranshu Ranjan ;
Tian, Jing .
BIOMEDICAL SIGNAL PROCESSING AND CONTROL, 2020, 57
[36]   Fusion of Hyperspectral and Panchromatic Images Using Generative Adversarial Network and Image Segmentation [J].
Dong, Wenqian ;
Yang, Yufei ;
Qu, Jiahui ;
Xie, Weiying ;
Li, Yunsong .
IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, 2022, 60
[37]   Semisupervised Multiscale Generative Adversarial Network for Semantic Segmentation of Remote Sensing Image [J].
Wang, Jiaqi ;
Liu, Bing ;
Zhou, Yong ;
Zhao, Jiaqi ;
Xia, Shixiong ;
Yang, Yuancan ;
Zhang, Man ;
Ming, Liu Ming .
IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, 2022, 19
[38]   ISRnet: Compressed Image Inpainting Based on Generative Adversarial Network [J].
Huang, Junjian ;
Zheng, Mao ;
Li, Zhizhang ;
He, Xing ;
Wen, Shiping .
IEEE TRANSACTIONS ON EMERGING TOPICS IN COMPUTATIONAL INTELLIGENCE, 2025, 9 (04) :2743-2753
[39]   A CONTEXT-BASED NETWORK FOR REFERRING IMAGE SEGMENTATION [J].
Li, Xinyu ;
Liu, Yu ;
Xu, Kaiping ;
Zhao, Zhehuan ;
Liu, Sipei .
2020 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 2020, :1436-1440
[40]   Modified Perceptual Cycle Generative Adversarial Network-Based Image Enhancement for Improving Accuracy of Low Light Image Segmentation [J].
Cho, Se Woon ;
Baek, Na Rae ;
Koo, Ja Hyung ;
Park, Kang Ryoung .
IEEE ACCESS, 2021, 9 :6296-6324
12345