Multi-modal neural networks with multi-scale RGB-T fusion for semantic segmentation

被引：19

作者：

Lyu, Y. ^{[1
]}

Schiopu, I. ^{[1
]}

Munteanu, A. ^{[1
]}

机构：

[1] Vrije Univ Brussel, Dept Elect & Informat, Pl Laan 2, B-1050 Brussels, Belgium

来源：

ELECTRONICS LETTERS | 2020年 / 56卷 / 18期

关键词：

infrared imaging; image fusion; feature extraction; neural nets; image resolution; image segmentation; encoding; learning (artificial intelligence); multimodal neural networks; multiscale RGB-T fusion; semantic segmentation; thermal images; neural network design; multimodal fusion; multiresolution patch processing; decoder module; thermal features; separate encoder streams;

D O I：

10.1049/el.2020.1635

中图分类号：

TM [电工技术]; TN [电子技术、通信技术];

学科分类号：

0808 ; 0809 ;

摘要：

A novel deep-learning-based method for semantic segmentation of RGB and thermal images is introduced. The proposed method employs a novel neural network design for multi-modal fusion based on multi-resolution patch processing. A novel decoder module is introduced to fuse the RGB and thermal features extracted by separate encoder streams. Experimental results on synthetic and real-world data demonstrate the efficiency of the proposed method compared with state-of-the-art methods.

引用

页码：920 / 922

页数：3

共 12 条

[1] DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs [J].

Chen, Liang-Chieh ;

Papandreou, George ;

Kokkinos, Iasonas ;

Murphy, Kevin ;

Yuille, Alan L. .

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 2018, 40 (04) :834-848

[2] The Cityscapes Dataset for Semantic Urban Scene Understanding [J].

Cordts, Marius ;

Omran, Mohamed ;

Ramos, Sebastian ;

Rehfeld, Timo ;

Enzweiler, Markus ;

Benenson, Rodrigo ;

Franke, Uwe ;

Roth, Stefan ;

Schiele, Bernt .

2016 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 2016, :3213-3223

[3]

Corsi C., 1995, Microsystem Technologies, V1, P149

[4]

Ha Q, 2017, IEEE INT C INT ROBOT, P5108, DOI 10.1109/IROS.2017.8206396

[5] Deep Residual Learning for Image Recognition [J].

He, Kaiming ;

Zhang, Xiangyu ;

Ren, Shaoqing ;

Sun, Jian .

2016 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 2016, :770-778

[6]

Hwang S, 2015, PROC CVPR IEEE, P1037, DOI 10.1109/CVPR.2015.7298706

[7] Image-to-Image Translation with Conditional Adversarial Networks [J].

Isola, Phillip ;

Zhu, Jun-Yan ;

Zhou, Tinghui ;

Efros, Alexei A. .

30TH IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2017), 2017, :5967-5976

[8] RTFNet: RGB-Thermal Fusion Network for Semantic Segmentation of Urban Scenes [J].

Sun, Yuxiang ;

Zuo, Weixun ;

Liu, Ming .

IEEE ROBOTICS AND AUTOMATION LETTERS, 2019, 4 (03) :2576-2583

[9] High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs [J].

Wang, Ting-Chun ;

Liu, Ming-Yu ;

Zhu, Jun-Yan ;

Tao, Andrew ;

Kautz, Jan ;

Catanzaro, Bryan .

2018 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 2018, :8798-8807

[10] Aggregated Residual Transformations for Deep Neural Networks [J].

Xie, Saining ;

Girshick, Ross ;

Dollar, Piotr ;

Tu, Zhuowen ;

He, Kaiming .

30TH IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2017), 2017, :5987-5995

← 1 2 →