Visible-Thermal Image Object Detection via the Combination of Illumination Conditions and Temperature Information

被引:26
作者
Zhou, Hang [1 ]
Sun, Min [1 ]
Ren, Xiang [1 ]
Wang, Xiuyuan [1 ]
机构
[1] Peking Univ, Inst Remote Sensing & GIS, Beijing 100871, Peoples R China
关键词
object detection; multi-spectral fusion; visible and thermal images; RetinaNet; illumination conditions; dynamic weight fusion; temperature information; a priori knowledge; FUSION; CNN;
D O I
10.3390/rs13183656
中图分类号
X [环境科学、安全科学];
学科分类号
08 ; 0830 ;
摘要
Object detection plays an important role in autonomous driving, disaster rescue, robot navigation, intelligent video surveillance, and many other fields. Nonetheless, visible images are poor under weak illumination conditions, and thermal infrared images are noisy and have low resolution. Consequently, neither of these two data sources yields satisfactory results when used alone. While some scholars have combined visible and thermal images for object detection, most did not consider the illumination conditions and the different contributions of diverse data sources to the results. In addition, few studies have made use of the temperature characteristics of thermal images. Therefore, in the present study, visible and thermal images are utilized as the dataset, and RetinaNet is used as the baseline to fuse features from different data sources for object detection. Moreover, a dynamic weight fusion method, which is based on channel attention according to different illumination conditions, is used in the fusion component, and the channel attention and a priori temperature mask (CAPTM) module is proposed; the CAPTM can be applied to a deep learning network as a priori knowledge and maximizes the advantage of temperature information from thermal images. The main innovations of the present research include the following: (1) the consideration of different illumination conditions and the use of different fusion parameters for different conditions in the feature fusion of visible and thermal images; (2) the dynamic fusion of different data sources in the feature fusion of visible and thermal images; (3) the use of temperature information as a priori knowledge (CAPTM) in feature extraction. To a certain extent, the proposed methods improve the accuracy of object detection at night or under other weak illumination conditions and with a single data source. Compared with the state-of-the-art (SOTA) method, the proposed method is found to achieve superior detection accuracy with an overall mean average precision (mAP) improvement of 0.69%, including an AP improvement of 2.55% for the detection of the Person category. The results demonstrate the effectiveness of the research methods for object detection, especially temperature information-rich object detection.
引用
收藏
页数:20
相关论文
共 50 条
  • [11] Object detection based on combination of local and spatial information
    Qinkun Xiao 1
    2.Fujitsu Research & Development Center Co.
    3.Department of Automation
    JournalofSystemsEngineeringandElectronics, 2011, 22 (04) : 715 - 720
  • [12] Object Detection in Visible and Infrared missile borne fusion image
    Xue, Song
    Liu, Yongfeng
    Xu, Chao
    Li, Jun
    2022 INTERNATIONAL CONFERENCE ON IMAGE PROCESSING, COMPUTER VISION AND MACHINE LEARNING (ICICML), 2022, : 19 - 23
  • [13] Combination of colour and thermal sensors for enhanced object detection
    St-Laurent, Louis
    Maldague, Xavier
    Prevost, Donald
    2007 PROCEEDINGS OF THE 10TH INTERNATIONAL CONFERENCE ON INFORMATION FUSION, VOLS 1-4, 2007, : 283 - +
  • [14] Thermal image processing-based intelligent technique for object detection
    Sayantan Nath
    C. Mala
    Signal, Image and Video Processing, 2022, 16 : 1631 - 1639
  • [15] Thermal image processing-based intelligent technique for object detection
    Nath, Sayantan
    Mala, C.
    SIGNAL IMAGE AND VIDEO PROCESSING, 2022, 16 (06) : 1631 - 1639
  • [16] Local Brightness Normalization for Image Classification and Object Detection Robust to Illumination Changes
    Lu, Yanshuo
    Tanaka, Masayuki
    Kawakami, Rei
    Okutomi, Masatoshi
    IMAGE AND VIDEO TECHNOLOGY, PSIVT 2023, 2024, 14403 : 56 - 68
  • [17] CDYL for infrared and visible light image dense small object detection
    Wu, Huixin
    Zhu, Yang
    Li, Shuqi
    SCIENTIFIC REPORTS, 2024, 14 (01)
  • [18] CDYL for infrared and visible light image dense small object detection
    Huixin Wu
    Yang Zhu
    Shuqi Li
    Scientific Reports, 14
  • [19] Object detection via deeply exploiting depth information
    Hou, Saihui
    Wang, Zilei
    Wu, Feng
    NEUROCOMPUTING, 2018, 286 : 58 - 66
  • [20] Object Detection to Evaluate Image-to-Image Translation on Different Road Conditions
    Sudo, Fumiya
    Hashimoto, Yoshihiro
    Lisi, Giuseppe
    INTELLIGENT HUMAN SYSTEMS INTEGRATION 2020, 2020, 1131 : 143 - 149