Hyperspectral Image Super-Resolution Network Based on Cross-Scale Nonlocal Attention

Hyperspectral image (HSI) super-resolution generally means the fusion of low-spatial-resolution HSI (LRHSI) and high-spatial-resolution multispectral/panchromatic image (HRMPI) to get high-spatial-resolution HSI (HRHSI). Existing fusion methods have not sufficiently considered the huge spectral and spatial resolution difference between the LRHSI and HRMPI. In addition, most deep learning (DL)-based methods that adopt the convolutional neural network (CNN) structure are limited by its local feature learning, and it is difficult to exploit the global dependence of image features. To fully adapt to the huge modality difference between LRHSI and HRMPI and release the limitation of local feature learning, we design the cross spectral-scale and shift-window-based cross spatial-scale nonlocal attention network (CSSNet) to effectively fuse the LRHSI and HRMPI. These two networks could explicitly learn the spectral and spatial correlations between two input images. These correlations are then used to reconstruct the HRHSI feature, which makes the obtained HRHSI feature to maintain the spectral and spatial feature consistency with the input images. Finally, a "feature aggregation module" is designed to aggregate the image features from these two networks and output the fused HRHSI. Extensive experimental results on both HM-fusion [fusion with multispectral (MSI)] and HP-fusion (fusion with panchromatic (PAN) image) tasks demonstrate CSSNet's state-of-the-art (SOTA) performance compared to other fusion methods. The codes could be available at https://github.com/rs-lsl/CSSNet.