Realtime Multispectral Pedestrian Detection With Visible and Far-Infrared Under Ambient Temperature Changing

In recent intelligent transportation systems (ITS), it is important to recognize pedestrians and avoid collisions. Various sensors are used to detect pedestrians, and some research on pedestrian detection uses a visible light (RGB) camera and a far-infrared (FIR) camera. FIR cameras are significantly affected by ambient temperatures such as summer and winter. However, few studies have focused on this property when evaluating pedestrian detection accuracy. Therefore, this paper investigates the effect of temperature change in real-time multispectral pedestrian detection. We created an original dataset with three subsets, Hot, Intermediate, and Cold, and evaluated temperature effects by changing the subsets during training and testing. We first evaluated YOLOv8s-4ch, which simply extended the input layer of YOLOv8 from 3 channels of RGB to 4 channels of RGB-FIR. To further improve detection performance, we built a new model called YOLOv8s-2stream. This model has two backbones for RGB and FIR, and fuses their feature maps in each resolution. We found that the model trained on a specific temperature subset dropped the test accuracy in other subsets. On the other hand, when training using a Mix set covering all temperature sets (Hot, Inter., Cold), the model achieved the highest accuracy through all conditions. Moreover, our YOLOv8s-2stream has improved by 3.9 points of accuracy (AP@0.5:0.95) compared to YOLOv8s-4ch, and achieved 73 FPS inference speed on Jetson.