Coupled Matrix Factorization Constrained DeepHyperspectral and Multispectral Image Fusion

The application of convolutional neural net-works (CNNs) has yielded remarkable outcomes in the fusionof hyperspectral and multispectral images (HSI+MSI). How-ever, most existing researches design black-box models forthe direct reconstruction of high-resolution images fromlow-resolution images, which cannot theoretically guaranteethe fusion mechanism throughout the network flow, thuslimiting the restoration accuracy. This article proposes anovel coupled matrix factorization (CMF) constrained deepinterpretable network for HSI+MSI fusion, termed as CMF-FUSnet. Specifically, the iterative process of CMF is unfoldedinto a two-branch network interwoven with multiple denoisermodules and matrix factorization modules. In each itera-tion, the CMF-encoded two-branch sub-network alternatelydecomposes HSI and MSI to estimate their abundance andendmember matrices, respectively. High-resolution HSI canbe obtained by multiplying the endmembers extracted fromthe HSI and the abundances extracted from the MSI. The benefit is that the proposed CMF-FUSnet breaks through theblack-box operation mode while adopting an end-to-end data-driven model, realizes the embedding of physical meaning,and improves the generalization of the model. Numerical experiments show that our proposed CMF-FUSnet comparesfavorably with both state-of-the-art model-driven and data-driven fusion methods in terms of visual analysis and qualityassessment.