Knowledge-Guided Multi-Task Network for Remote Sensing Imagery

Semantic segmentation and height estimation tasks in remote sensing imagery exhibit distinctive characteristics, including scale sensitivity, category imbalance, and insufficient fine details. Recent approaches have leveraged multi-task learning methods to jointly predict these tasks along with auxiliary tasks, such as edge detection, to improve the accuracy of fine-grained details. However, most approaches only acquire knowledge from auxiliary tasks, disregarding the inter-task knowledge guidance across all tasks. To address these challenges, we propose KMNet, a novel architecture referred to as a knowledge-guided multi-task network, which can be applied to different primary and auxiliary task combinations. KMNet employs a multi-scale methodology to extract feature information from the input image. Subsequently, the architecture incorporates the multi-scale knowledge-guided fusion (MKF) module, which is designed to generate a comprehensive knowledge bank serving as a resource for guiding the feature fusion process. The knowledge-guided fusion feature is then utilized to generate the final predictions for the primary tasks. Comprehensive experiments conducted on two publicly available remote sensing datasets, namely the Potsdam dataset and the Vaihingen dataset, demonstrate the effectiveness of the proposed method in achieving impressive performance on both semantic segmentation and height estimation tasks. Codes, pre-trained models, and more results will be publicly available.