Multispectral Pedestrian Detection with Visible and Far-infrared Images Under Drifting Ambient Light and Temperature

The contrast between pedestrians and the background in visible (RGB) images greatly depends on the ambient light, while in far-infrared (FIR) images, it depends on the ambient temperature. Although existing open datasets for RGB-FIR combined images consider ambient light differences between daytime and nighttime, these datasets do not take into account temperature variations between hot and cold conditions. In this paper, we prepare new datasets that include both environmental differences, and we evaluate them using a multispectral detection system. Our results show that RGB-FIR detection achieves higher accuracy with an average precision (AP) compared with RGB detection when ambient light changes. Furthermore, the results indicate that RGB-FIR detection outperforms FIR detection when ambient temperature changes. Finally, the result of the cross-validate experiment shows that the RGB-FIR model trained in intermediate temperature conditions decreases by 48.3 points and 24.7 points under the hot and cold conditions even though it achieved a high AP: 83.0 % under the training condition. The results suggest all environmental variations should be included when we train the RGB-FIR model for pedestrian detection.