MJ-GAN: Generative Adversarial Network with Multi-Grained Feature Extraction and Joint Attention Fusion for Infrared and Visible Image Fusion

被引:3
|
作者
Yang, Danqing [1 ]
Wang, Xiaorui [1 ]
Zhu, Naibo [2 ]
Li, Shuang [2 ]
Hou, Na [2 ]
机构
[1] Xidian Univ, Sch Optoelect Engn, Xian 710071, Peoples R China
[2] PLA Acad Mil Sci, Res Inst Syst Engn, Beijing 100091, Peoples R China
关键词
infrared and visible image fusion (IVIF); multi-scale feature extraction (MFE); joint attention fusion (JAF); generative adversarial network (GAN); self-attention mechanism (SAM); QUALITY MEASURES; TRANSFORM; FOCUS;
D O I
10.3390/s23146322
中图分类号
O65 [分析化学];
学科分类号
070302 ; 081704 ;
摘要
The challenging issues in infrared and visible image fusion (IVIF) are extracting and fusing as much useful information as possible contained in the source images, namely, the rich textures in visible images and the significant contrast in infrared images. Existing fusion methods cannot address this problem well due to the handcrafted fusion operations and the extraction of features only from a single scale. In this work, we solve the problems of insufficient information extraction and fusion from another perspective to overcome the difficulties in lacking textures and unhighlighted targets in fused images. We propose a multi-scale feature extraction (MFE) and joint attention fusion (JAF) based end-to-end method using a generative adversarial network (MJ-GAN) framework for the aim of IVIF. The MFE modules are embedded in the two-stream structure-based generator in a densely connected manner to comprehensively extract multi-grained deep features from the source image pairs and reuse them during reconstruction. Moreover, an improved self-attention structure is introduced into the MFEs to enhance the pertinence among multi-grained features. The merging procedure for salient and important features is conducted via the JAF network in a feature recalibration manner, which also produces the fused image in a reasonable manner. Eventually, we can reconstruct a primary fused image with the major infrared radiometric information and a small amount of visible texture information via a single decoder network. The dual discriminator with strong discriminative power can add more texture and contrast information to the final fused image. Extensive experiments on four publicly available datasets show that the proposed method ultimately achieves phenomenal performance in both visual quality and quantitative assessment compared with nine leading algorithms.
引用
收藏
页数:29
相关论文
共 50 条
  • [1] AT-GAN: A generative adversarial network with attention and transition for infrared and visible image fusion
    Rao, Yujing
    Wu, Dan
    Han, Mina
    Wang, Ting
    Yang, Yang
    Lei, Tao
    Zhou, Chengjiang
    Bai, Haicheng
    Xing, Lin
    INFORMATION FUSION, 2023, 92 : 336 - 349
  • [2] MFAGAN: A multiscale feature-attention generative adversarial network for infrared and visible image fusion
    Tang, Xuanji
    Zhao, Jufeng
    Cui, Guangmang
    Tian, Haijun
    Shi, Zhen
    Hou, Changlun
    INFRARED PHYSICS & TECHNOLOGY, 2023, 133
  • [3] Infrared and visible image fusion using a feature attention guided perceptual generative adversarial network
    Chen Y.
    Zheng W.
    Shin H.
    Journal of Ambient Intelligence and Humanized Computing, 2023, 14 (07) : 9099 - 9112
  • [4] FusionGAN: A generative adversarial network for infrared and visible image fusion
    Ma, Jiayi
    Yu, Wei
    Liang, Pengwei
    Li, Chang
    Jiang, Junjun
    INFORMATION FUSION, 2019, 48 : 11 - 26
  • [5] Infrared and Visible Image Fusion with a Generative Adversarial Network and a Residual Network
    Xu, Dongdong
    Wang, Yongcheng
    Xu, Shuyan
    Zhu, Kaiguang
    Zhang, Ning
    Zhang, Xin
    APPLIED SCIENCES-BASEL, 2020, 10 (02):
  • [6] EMA-GAN: A Generative Adversarial Network for Infrared and Visible Image Fusion with Multiscale Attention Network and Expectation Maximization Algorithm
    Xi, Xiuliang
    Jin, Xin
    Jiang, Qian
    Lin, Yu
    Zhou, Wei
    Guo, Lei
    ADVANCED INTELLIGENT SYSTEMS, 2023, 5 (11)
  • [7] EMA-GAN: A Generative Adversarial Network for Infrared and Visible Image Fusion with Multiscale Attention Network and Expectation Maximization Algorithm
    Xi, Xiuliang
    Jin, Xin
    Jiang, Qian
    Lin, Yu
    Zhou, Wei
    Guo, Lei
    ADVANCED INTELLIGENT SYSTEMS, 2023,
  • [8] DFPGAN: Dual fusion path generative adversarial network for infrared and visible image fusion
    Yi, Shi
    Li, Junjie
    Yuan, Xuesong
    INFRARED PHYSICS & TECHNOLOGY, 2021, 119
  • [9] Infrared and visible image fusion based on edge-preserving and attention generative adversarial network
    Zhu Wen-Qing
    Tang Xin-Yi
    Zhang Rui
    Chen Xiao
    Miao Zhuang
    JOURNAL OF INFRARED AND MILLIMETER WAVES, 2021, 40 (05) : 696 - 708
  • [10] Laplacian Pyramid Generative Adversarial Network for Infrared and Visible Image Fusion
    Yin, Haitao
    Xiao, Jinghu
    IEEE SIGNAL PROCESSING LETTERS, 2022, 29 : 1988 - 1992