Deep spatial-spectral fusion transformer for remote sensing pansharpening

Pansharpening is the task which reconstructs spatial-spectral properties during the fusion of high-resolution panchromatic (PAN) with low-resolution multi-spectral (LR-MS) images, to generate a high-resolution multi- spectral (HR-MS) image. Recent approaches typically model spatial and spectral properties and fuse them using end-to-end deep learning networks, which fail to take their crucial task-related relationship priors into consideration. In this paper, we propose a novel deep spatial-spectral fusion Transformer (SSFT) inspired by two crucial task-related findings: (i) spatial property-related prior, i.e., PAN image itself can provide enough spatial property to reconstruct the required spatial property of target HR-MS image; and (ii) spectral property-related prior, i.e., both LR-MS and PAN should be involved in the process of modeling spectral property for target HR-MS image. Specifically, our approach consists of three novel blocks: the Fourier- guided spectral reconstruction (FGSR) block innovatively applies complex feature interaction strategy to Fourier representations2 of LR-MS and PAN images, for reconstructing the amplitude manifesting the spectral property for the target HR-MS; the Top-k spatial reconstruction (TKSR) block exploits the Top-k selection strategy to select the most relevant spatial regions from PAN image, for modeling the required spatial property of the target HR-MS; and the spatial-spectral fusion (SSF) block re-weights value (V) matrix of FGSR's output feature according to the TKSR's output feature, thus achieving a seamless integration of spatial and spectral properties. Extensive experiments show that our SSFT significantly outperforms state-of-the-art (SOTA) methods on widely-used WorldView-3, QuickBird and GaoFen-2 datasets. Our code is available at https://github.com/Florina2333/SSFT.