RTFNet: RGB-Thermal Fusion Network for Semantic Segmentation of Urban Scenes

被引:312
作者
Sun, Yuxiang [1 ]
Zuo, Weixun [1 ]
Liu, Ming [1 ]
机构
[1] Hong Kong Univ Sci & Technol, Dept Elect & Comp Engn, Kowloon, Clear Water Bay, Hong Kong, Peoples R China
基金
中国国家自然科学基金;
关键词
Semantic Segmentation; Urban Scenes; Deep Neural Network; Thermal Images; Information Fusion; MOTION REMOVAL; D SLAM;
D O I
10.1109/LRA.2019.2904733
中图分类号
TP24 [机器人技术];
学科分类号
080202 ; 1405 ;
摘要
Semantic segmentation is a fundamental capability for autonomous vehicles. With the advancements of deep learning technologies, many effective semantic segmentation networks have been proposed in recent years. However, most of them are designed using RGB images from visible cameras. The quality of RGB images is prone to be degraded under unsatisfied lighting conditions, such as darkness and glares of oncoming headlights, which imposes critical challenges for the networks that use only RGB images. Different from visible cameras, thermal imaging cameras generate images using thermal radiations. They are able to see under various lighting conditions. In order to enable robust and accurate semantic segmentation for autonomous vehicles, we take the advantage of thermal images and fuse both the RGB and thermal information in a novel deep neural network. The main innovation of this letter is the architecture of the proposed network. We adopt the encoder-decoder design concept. ResNet is employed for feature extraction and a new decoder is developed to restore the feature map resolution. The experimental results prove that our network outperforms the state of the arts.
引用
收藏
页码:2576 / 2583
页数:8
相关论文
共 29 条
[1]  
[Anonymous], P 3 INT C LEARNING R
[2]  
[Anonymous], PROC CVPR IEEE
[3]  
[Anonymous], IEEE T PATTERN ANAL
[4]  
[Anonymous], 2012, NEURAL NETWORKS TRIC
[5]  
[Anonymous], IEEE Transactions on Automation Science and Engineering
[6]  
[Anonymous], IEEE T AUTOM SCI ENG
[7]  
[Anonymous], 2017, IEEE T PATTERN ANAL, DOI DOI 10.1109/TPAMI.2016.2644615
[8]   Robust Intrinsic and Extrinsic Calibration of RGB-D Cameras [J].
Basso, Filippo ;
Menegatti, Emanuele ;
Pretto, Alberto .
IEEE TRANSACTIONS ON ROBOTICS, 2018, 34 (05) :1315-1332
[9]   Obstacle Detection System for Small UAVs using ADS-B and Thermal Imaging [J].
Carrio, Adrian ;
Lin, Yucong ;
Saripalli, Srikanth ;
Campoy, Pascual .
JOURNAL OF INTELLIGENT & ROBOTIC SYSTEMS, 2017, 88 (2-4) :583-595
[10]   DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs [J].
Chen, Liang-Chieh ;
Papandreou, George ;
Kokkinos, Iasonas ;
Murphy, Kevin ;
Yuille, Alan L. .
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 2018, 40 (04) :834-848