Multimodal-Aware Fusion Network for Referring Remote Sensing Image Segmentation

Referring remote sensing image segmentation (RRSIS) is a novel visual task in remote sensing images segmentation, which aims to segment objects based on a given text description, with great significance in practical application. Previous studies fuse visual and linguistic modalities by explicit feature interaction, which fail to effectively excavate useful multimodal information from dual-branch encoder. In this letter, we design a multimodal-aware fusion network (MAFN) to achieve fine-grained alignment and fusion between the two modalities. We propose a correlation fusion module (CFM) to enhance multiscale visual features by introducing adaptive noise in transformer, and integrate cross-modal aware features. In addition, MAFN employs multiscale refinement convolution (MSRC) to adapt to the various orientations of objects at different scales to boost their representation ability to enhances segmentation accuracy. Extensive experiments have shown that MAFN is significantly more effective than the state of the art (SOTA) on RRSIS-D datasets. The source code is available at https://github.com/Roaxy/MAFN.