TSTR: A Real-Time RGB-Thermal Semantic Segmentation Model with Multimodal Fusion Transformers

Semantic segmentation plays a crucial role in scene understanding, particularly in the context of autonomous driving. Traditional methods mainly rely on single-modal RGB images, which perform well only under good lighting conditions during the daytime and poorly in challenging lighting conditions like nighttime. Recent research has started integrating thermal images generated by thermal cameras as an additional modality into deep neural networks, known as RGB-T semantic segmentation. However, existing approaches are limited by the limited receptive field of CNNs and often rely on complex fusion modules to improve accuracy, neglecting the computational time of the model. To address these issues, we propose a real-time multimodal fusion network (TSTR) for RGB-T semantic segmentation. Specifically, taking advantage of the global self-attention mechanism and long-range dependency modeling of transformers, we design a dual-branch encoder based on transformers to extract RGB and thermal features separately. To ensure real-time performance, we introduce more efficient self-attention blocks and employ a lightweight MLP decoder to aggregate information from different levels for dense predictions. Additionally, we employ a simple element-wise addition operation for multimodal feature fusion. Extensive quantitative and qualitative experiments demonstrate that the proposed network outperforms several state-of-the-art real-time RGB-T semantic segmentation methods. It achieves an mIoU of 56.2% on benchmark datasets, running in real-time at a speed of 59 FPS with a 480x640 input on an NVIDIA GeForce GTX 2080 Ti GPU. This method is significantly faster than existing networks while achieving better segmentation accuracy.