A Foreground-Driven Fusion Network for Gully Erosion Extraction Utilizing UAV Orthoimages and Digital Surface Models

Unmanned aerial vehicle (UAV) orthoimages and digital surface models (DSMs) can provide valuable insights for semantic segmentation methods in comprehending gully erosion (GE) from diverse perspectives. While the integration of these two modalities has the potential to improve the GE extraction performance, the extent of enhancement primarily depends on the quality of modality-specific features and the synergistic fusion manner employed for integrating features from both modalities. Toward this end, we propose a novel multimodal segmentation method, which is called foreground-driven fusion network (FFNet). Guided by the prototypes of foreground objects (i.e., gullies), the network effectively tackles the challenges from the modality itself and between different modalities, ultimately achieving high-quality GE extraction results. Specifically, a foreground prototype sampling (FPS) module is first devised for precisely sampling foreground prototypes related to gullies from two modalities. Then, a local-global hybrid purification (LHP) module is proposed to effectively mitigate the erroneous activation within each modality at multiple dimensions by leveraging foreground prototypes. Finally, a multimodal foreground synergy (MFS) module is introduced to further activate foreground features and facilitate full complementarity between multimodal foreground features. To validate our network, a comprehensive multimodal dataset for GE extraction is constructed based on UAV orthoimages and DSMs from northeastern China. Furthermore, a public road extraction dataset is employed to evaluate the generalizability of this network. In the experiments conducted on these two datasets, the proposed FFNet exhibits obvious superiority, outperforming the second-best method with an average improvement of 2.55% in terms of intersection over union (IoU) and 2.77% in terms of $F1$ -score. These experimental results not only demonstrate the practicality of FFNet in GE extraction tasks, but also highlight its significant advantage in similar road extraction tasks.