Sensor Fusion Operators for Multimodal 2D Object Detection

Autonomous driving requires effective capabilities to detect road objects in different environmental conditions. One promising solution to improve perception is to leverage multi-sensor fusion. This approach aims to combine various sensor streams in order to best integrate the information coming from the different sensors. Fusion operators are used to combine features from different modalities inside convolutional neural network architectures. In this study, we provide a framework for evaluating early fusion operators using different 2D object detection architectures. This comparative study includes element-wise addition and multiplication, feature concatenation, multi-modal factorized bilinear pooling, and bilaterally-guided fusion. We report quantitative results of the performance as well as an analysis of computational costs of these operators on different architectures.