Infrared and visible image fusion using a shallow CNN and structural similarity constraint

被引:13
作者
Li, Lei [1 ]
Xia, Zhaoqiang [2 ]
Han, Huijian [1 ]
He, Guiqing [2 ]
Roli, Fabio [2 ,3 ]
Feng, Xiaoyi [2 ]
机构
[1] Shandong Univ Finance & Econ, Sch Comp Sci & Technol, Jinan, Shandong, Peoples R China
[2] Northwestern Polytech Univ, Sch Elect & Informat, Xian, Shaanxi, Peoples R China
[3] Univ Cagliari, Dept Elect & Elect Engn, Cagliari, Italy
基金
中国国家自然科学基金;
关键词
learning (artificial intelligence); infrared imaging; image fusion; convolutional neural nets; convolutional layer; fused single image; visible image; structural similarity constraint; infrared image fusion method; training data; scarce reference images; multisource images; deep models; end-to-end shallow convolutional neural network; visible image fusion; shallow CNN; rectified linear unit function; structural similarity loss; pixel misalignment; SHEARLET TRANSFORM;
D O I
10.1049/iet-ipr.2020.0360
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
In recent years, image fusion methods based on deep networks have been proposed to combine infrared and visible images for achieving better fusion image. However, issues such as limited training data, scarce reference images and misalignment of multi-source images, still limit the fusion performance. To address these problems, we propose an end-to-end shallow convolutional neural network with structural constraints, which has only one convolutional layer to fuse infrared and visible images. Different from other methods, our proposed model requires less training data and reference images and is more robust to the misalignment of a couple of images. More specifically, the infrared image and the visible image are first provided as inputs to a convolutional layer to extract the information that should be fused; then, all feature maps are concatenated together and fed into a convolutional layer with one channel to obtain the fused image; finally, a structural similarity loss between the fused image and the input infrared and visible images is computed to update the network parameters and eliminate the effects of pixel misalignment. Extensive experiments show the effectiveness of our proposed method on fusion of infrared and visible images with the performance that outperforms the state-of-the-art methods.
引用
收藏
页码:3562 / 3571
页数:10
相关论文
共 42 条
  • [1] [Anonymous], 2015, IEEE I CONF COMP VIS, DOI DOI 10.1109/ICCV.2015.123
  • [2] Two-scale image fusion of visible and infrared images using saliency detection
    Bavirisetti, Durga Prasad
    Dhuli, Ravindra
    [J]. INFRARED PHYSICS & TECHNOLOGY, 2016, 76 : 52 - 64
  • [3] Ben Hamza A, 2005, INTEGR COMPUT-AID E, V12, P135
  • [4] Cao AX, 2016, DESTECH TRANS MAT, P78
  • [5] Dhyani S., 2016, SIGNAL IMAGE VIDEO P, V7, P1125
  • [6] Image Fusion With Cosparse Analysis Operator
    Gao, Rui
    Vorobyov, Sergiy A.
    Zhao, Hong
    [J]. IEEE SIGNAL PROCESSING LETTERS, 2017, 24 (07) : 943 - 947
  • [7] Glorot X., 2011, P 14 INT C ART INT S
  • [8] Haghighat M, 2014, I C APPL INF COMM TE, P424
  • [9] The problem of overfitting
    Hawkins, DM
    [J]. JOURNAL OF CHEMICAL INFORMATION AND COMPUTER SCIENCES, 2004, 44 (01): : 1 - 12
  • [10] Image fusion using multiscale edge-preserving decomposition based on weighted least squares filter
    Jiang, Yong
    Wang, Minghui
    [J]. IET IMAGE PROCESSING, 2014, 8 (03) : 183 - 190