Multi-feature decomposition and transformer-fusion: an infrared and visible image fusion network based on multi-feature decomposition and transformer

Most of the existing infrared and visible image fusion methods focus only on the extraction of detail and basic features of both images. However, these methods ignore the extraction of shallow common features, which causes the problem of weakening the ability of detail and basic feature extraction. Consequently, we propose an infrared and visible image fusion network based on multi-feature decomposition and transformer (MFDT-Fusion). The multi-feature decomposition is processed by three encoders. First, to fully extract the shallow common features of the images, multi-head gated block based on transformer improvement is proposed as the common feature encoder (CFE). Multi-head dilated attention is designed in CFE to aggregate information from different receptive fields of infrared and visible images. Then, the long-range information-capturing ability of the transformer is used to compensate for the limited receptive field problem of the convolutional neural network. The detail feature encoder and basic feature encoder are designed to decompose the two features of the images. Finally, the features are separately fused by detail and basic fusion layers. Experimental results on three public datasets show that MFDT-Fusion achieves better performance than state-of-the-art methods. (c) 2024 SPIE and IS&T