QIS-GAN: A Lightweight Adversarial Network With Quadtree Implicit Sampling for Multispectral and Hyperspectral Image Fusion

Multispectral and hyperspectral image fusion (MHIF) involves the fusion of high-spatial-resolution multispectral images (HR-MSIs) and low-spatial-resolution hyperspectral images (LR-HSIs) to generate high-spatial-resolution hyperspectral images (HR-HSIs) and has gained significant attention in the field of remote-sensing imaging. While CNN and Transformer models have shown effectiveness in MHIF, existing CNN-or Transformer-based algorithms are overburdened with model size, making it difficult to achieve an effective tradeoff between fusion accuracy and degree of lightweight. Recently, implicit neural representation (INR) has been proven good interpretability and the ability to exploit coordinate information in 2-D tasks. Nonetheless, INR-based fusion networks have certain limitations, such as the need for deeper super-resolution networks as shallow encoders, and insufficient representation capability on high upsampling ratios. To address these challenges, we present the quadtree implicit sampling (QIS), which employs a hierarchical sampling from the perspective of the quadtree, to enhance the capacity of the overall network. Furthermore, the remarkable design of QIS allows us to adopt a lightweight structure as the shallow encoder, greatly alleviating the network burden and achieving lightweight. Inspired by generative adversarial models, we incorporate QIS as a lightweight generator into the generative adversarial network (GAN) framework named QIS-GAN and leverage a discriminator to increase the fidelity of fused images. The results showcase the superior performance of QIS-GAN on the MHIF tasks with upsampling ratios of x4, x8, and x16, surpassing the state-of-the-art (SOTA) in several datasets. The code for our approach will be available at https://github.com/chunyuzhu/QIS-GAN.