PAG-Unet: multi-task dense scene understanding with pixel-attention-guided Unet

Multi-task dense scene understanding is a fundamental research area in computer vision (CV). By predicting pixels, perceiving, and reasoning about multiple related tasks, it improves both accuracy and data efficiency. However, it faces the challenge that some tasks may require more independent feature representations, and excessive sharing can lead to interference between tasks. To address this issue, we propose a novel Pixel-Attention-Guided Unet (PAG-Unet). PAG-Unet incorporates a Pixel-Attention-Guided Fusion module (PAG Fusion) and a Multi-Task Self-Attention module (MTSA) to enhance task-specific feature extraction and reduce task interference. PAG Fusion leverages the relationship between shallow and deep features by using task-specific deep features to calibrate the distribution of shared shallow features. This suppresses background noise and enhances semantic features, thereby fully extracting task-specific features for different tasks and achieving feature enhancement. MTSA considers both global and local spatial interactions for each task during task interactions, capturing task-specific information and compensating for the loss of crucial details, thus improving prediction accuracy for each task. Our method achieves superior multi-task performance on the New York University Depth v2(NYUD-v2) and PASCAL Visual Object Classes Context(PASCAL-Context) datasets, with most metrics significantly outperforming previous state-of-the-art methods. The code is available at https://github.com/UPLI-123/Pag-Unet.