Unsupervised Deep Tensor Network for Hyperspectral-Multispectral Image Fusion

被引:29
作者
Yang, Jingxiang [1 ,2 ]
Xiao, Liang [1 ,2 ,3 ]
Zhao, Yong-Qiang [4 ]
Chan, Jonathan Cheung-Wai [5 ]
机构
[1] Nanjing Univ Sci & Technol, Sch Comp Sci & Engn, Nanjing 210094, Peoples R China
[2] Nanjing Skyworth Inst Informat Technol Co Ltd, Nanjing 210012, Peoples R China
[3] Nanjing Univ Sci & Technol, Jiangsu Key Lab Spectral Imaging & Intelligent Sen, Nanjing 210094, Peoples R China
[4] Northwestern Polytech Univ, Sch Automat, Xian 710129, Peoples R China
[5] Vrije Univ Brussel, Dept Elect & Informat, B-1050 Brussels, Belgium
基金
中国国家自然科学基金;
关键词
Tensors; Degradation; Filtering; Codes; Feature extraction; Remote sensing; Dictionaries; Deep learning (DL); fusion; hyperspectral; tensor; DECOMPOSITION; RANK;
D O I
10.1109/TNNLS.2023.3266038
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Fusing low-resolution (LR) hyperspectral images (HSIs) with high-resolution (HR) multispectral images (MSIs) is a significant technology to enhance the resolution of HSIs. Despite the encouraging results from deep learning (DL) in HSI-MSI fusion, there are still some issues. First, the HSI is a multidimensional signal, and the representability of current DL networks for multidimensional features has not been thoroughly investigated. Second, most DL HSI-MSI fusion networks need HR HSI ground truth for training, but it is often unavailable in reality. In this study, we integrate tensor theory with DL and propose an unsupervised deep tensor network (UDTN) for HSI-MSI fusion. We first propose a tensor filtering layer prototype and further build a coupled tensor filtering module. It jointly represents the LR HSI and HR MSI as several features revealing the principal components of spectral and spatial modes and a sharing code tensor describing the interaction among different modes. Specifically, the features on different modes are represented by the learnable filters of tensor filtering layers, the sharing code tensor is learned by a projection module, in which a co-attention is proposed to encode the LR HSI and HR MSI and then project them onto the sharing code tensor. The coupled tensor filtering module and projection module are jointly trained from the LR HSI and HR MSI in an unsupervised and end-to-end way. The latent HR HSI is inferred with the sharing code tensor, the features on spatial modes of HR MSIs, and the spectral mode of LR HSIs. Experiments on simulated and real remote-sensing datasets demonstrate the effectiveness of the proposed method.
引用
收藏
页码:13017 / 13031
页数:15
相关论文
共 50 条
[41]   Iteratively Regularizing Hyperspectral and Multispectral Image Fusion With Framelets [J].
Shen, Xiangfei ;
Chen, Lihui ;
Liu, Haijun ;
Zhou, Xichuan ;
Bao, Wenxing ;
Tian, Ling ;
Vivione, Gemine ;
Chanussot, Jocelyn .
IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, 2025, 18 :5331-5346
[42]   Regularizing Hyperspectral and Multispectral Image Fusion by CNN Denoiser [J].
Dian, Renwei ;
Li, Shutao ;
Kang, Xudong .
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, 2021, 32 (03) :1124-1135
[43]   An Unsupervised Laplacian Pyramid Network for Radiometrically Accurate Data Fusion of Hyperspectral and Multispectral Imagery [J].
Huang, Sihan ;
Messinger, David W. .
IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, 2022, 60
[44]   Hyperspectral Image Classification With Deep Feature Fusion Network [J].
Song, Weiwei ;
Li, Shutao ;
Fang, Leyuan ;
Lu, Ting .
IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, 2018, 56 (06) :3173-3184
[45]   Hyperspectral Restoration and Fusion With Multispectral Imagery via Low-Rank Tensor-Approximation [J].
Liu, Na ;
Li, Lu ;
Li, Wei ;
Tao, Ran ;
Fowler, James E. ;
Chanussot, Jocelyn .
IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, 2021, 59 (09) :7817-7830
[46]   A VARIATIONAL FORMULATION FOR HYPERSPECTRAL AND MULTISPECTRAL IMAGE FUSION [J].
Mifdal, Jamila ;
Coll, Bartomeu ;
Duran, Joan .
2018 25TH IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 2018, :3328-3332
[47]   Mixed Noise-Oriented Hyperspectral and Multispectral Image Fusion [J].
Fu, Xiyou ;
Liang, Hong ;
Jia, Sen .
IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, 2023, 61
[48]   Stereo Cross-Attention Network for Unregistered Hyperspectral and Multispectral Image Fusion [J].
Guo, Yujuan ;
Fu, Xiyou ;
Xu, Meng ;
Jia, Sen .
IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, 2023, 61
[49]   Edge-Conditioned Feature Transform Network for Hyperspectral and Multispectral Image Fusion [J].
Zheng, Yuxuan ;
Li, Jiaojiao ;
Li, Yunsong ;
Guo, Jie ;
Wu, Xianyun ;
Shi, Yanzi ;
Chanussot, Jocelyn .
IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, 2022, 60
[50]   Hyperspectral and Multispectral Image Fusion With Functional Data Analysis Techniques [J].
Gongye, Lingxin ;
He, Shiyuan ;
Guo, Jianhua .
IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, 2025, 63