3-D Multiple Extended Object Tracking by Fusing Roadside Radar and Camera Sensors

Multiple object tracking is a key component of intelligent transportation systems. Most existing tracking methods rely on deep learning frameworks, requiring extensive high-quality 3-D annotated datasets, which limits their deployment in real-world scenarios. To address this problem, we present a novel 3-D multiple extended object tracking (MEOT) method by fusing radar and camera data with only pixel annotations. The fusion framework employs a joint detection and tracking (JDT) strategy based on random finite set (RFS) theory, and we propose a multimodal trajectory Poisson multi-Bernoulli (MTPMB) tracker to incorporate multimodal data from radar and camera for the first time. In the update stage, we propose a novel backprojection method and its corresponding visual measurement model based on a mixture density network (MDN) that automatically learns the transformation from pixel coordinates to 3-D road surface, allowing visual measurements to be accurately backprojected to 3-D space. Additionally, we introduce a doped association method that uses visual measurements to assist in splitting radar measurements from nearby objects and establish a correlation between radar and visual measurements. Experimental results demonstrate the effectiveness and superiority of the proposed method compared to several state-of-the-art 3-D MEOT methods without 3-D annotations. Code is available at https://github.com/RadarCameraFusionTeam-BUPT/ES-MEOT.