Coupling Local–Nonlocal Feature Representation for SAR and Multispectral Image Fusion

被引:0
作者
Zhu, Jiajia [1 ]
Liang, Hongbo [1 ]
Yang, Xuezhi [1 ]
Yang, Xiangyu [1 ]
机构
[1] Hefei Univ Technol, Sch Comp Sci & Informat, Key Lab Ind Safety & Emergency Technol Anhui, Hefei 230009, Peoples R China
关键词
Image fusion; Feature extraction; Mutual information; Transforms; Transformers; Semantics; Satellites; Convolutional neural network (CNN); paired tokens; synthetic aperture radar (SAR) and multispectral (MS) image fusion; vision transformer (ViT);
D O I
10.1109/LGRS.2024.3404616
中图分类号
P3 [地球物理学]; P59 [地球化学];
学科分类号
0708 ; 070902 ;
摘要
In this letter, we propose the CLN-Fusion, a novel hybrid fusion approach that leverages the merits of convolutional neural networks (CNNs) and vision transformers (ViTs) to couple local-nonlocal feature representations between synthetic aperture radar (SAR) and multispectral (MS) images. Specifically, we construct a paired token projection (PTP) to match the observation scenario content consistency of the two. Meanwhile, in terms of merging the complementary features between structures in SAR images and textures in MS images, we establish the pyramid CNN and ViT branches that assemble two pure feature volumes with convolutional inductive biases and nonlocal statistical correlation, respectively, into a mixed one. Furthermore, our CLN-Fusion maintains semantic alignment by maximizing mutual information throughout the PTP. Extensive experiments validate the superiority of the CLN-Fusion in terms of quantitative metrics, achieving SAR/MS image fusion under three scenarios from Sentinel-1 and Landsat8 data. With peak signal-noise ratios (PSNRs) of 33.1565, 30.9815, and 29.9821, showcasing the utmost fusion performance in contrast to other state-of-the-art (SOTA) methods. The codes of this work will be available at https://github.com/Blueseatear/CLN-Fusion.
引用
收藏
页数:5
相关论文
共 22 条
[1]   SAR and Multispectral Image Fusion Using Generalized IHS Transform Based on Trous Wavelet and EMD Decompositions [J].
Chen, Shaohui ;
Zhang, Renhua ;
Su, Hongbo ;
Tian, Jing ;
Xia, Jun .
IEEE SENSORS JOURNAL, 2010, 10 (03) :737-745
[2]   Self-Supervised SAR-Optical Data Fusion of Sentinel-1/-2 Images [J].
Chen, Yuxing ;
Bruzzone, Lorenzo .
IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, 2022, 60
[3]  
Dosovitskiy A, 2021, Arxiv, DOI arXiv:2010.11929
[4]   Multispectral and SAR Image Fusion for Multiscale Decomposition Based on Least Squares Optimization Rolling Guidance Filtering [J].
Gong, Xunqiang ;
Hou, Zhaoyang ;
Wan, Yuting ;
Zhong, Yanfei ;
Zhang, Meng ;
Lv, Kaiyun .
IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, 2024, 62 :1-20
[5]   A Fusion Method of Optical Image and SAR Image Based on Dense-UGAN and Gram-Schmidt Transformation [J].
Kong, Yingying ;
Hong, Fang ;
Leung, Henry ;
Peng, Xiangyang .
REMOTE SENSING, 2021, 13 (21)
[6]   Pixel level fusion techniques for SAR and optical images: A review [J].
Kulkarni, Samadhan C. ;
Rege, Priti P. .
INFORMATION FUSION, 2020, 59 :13-29
[7]   Fusion of SAR and Optical Image for Sea Ice Extraction [J].
Li, Wanwu ;
Liu, Lin ;
Zhang, Jixian .
JOURNAL OF OCEAN UNIVERSITY OF CHINA, 2021, 20 (06) :1440-1450
[8]   MAFusion: Multiscale Attention Network for Infrared and Visible Image Fusion [J].
Li, Xiaoling ;
Chen, Houjin ;
Li, Yanfeng ;
Peng, Yahui .
IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, 2022, 71
[9]   FDFNet: A Fusion Network for Generating High-Resolution Fully PolSAR Images [J].
Lin, Liupeng ;
Shen, Huanfeng ;
Li, Jie ;
Yuan, Qiangqiang .
IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, 2022, 19
[10]   COMPARATIVE ANALYSIS OF PIXEL LEVEL FUSION ALGORITHMS IN HIGH RESOLUTION SAR AND OPTICAL IMAGE FUSION [J].
Liu, Huiyu ;
Ye, Yuanxin ;
Zhang, Jiacheng ;
Yang, Chao ;
Zhao, Yangang .
2022 IEEE INTERNATIONAL GEOSCIENCE AND REMOTE SENSING SYMPOSIUM (IGARSS 2022), 2022, :2829-2832