nuScenesComplex: A More Rigorous Evaluation Framework for End-to-End Autonomous Driving Planning

The end-to-end autonomous driving planning task poses a significant challenge due to its inherent complexities, especially when existing datasets are inadequate for evaluating models' performance properly. While prior research extensively employs the nuScenes dataset for open-loop training and evaluation, its predominantly trivial scenes fail to effectively measure model performance in complex and dangerous scenarios. This study conducts a thorough analysis of the nuScenes evaluation set and proposes a specialized sub-dataset that spotlights challenging scenarios for more rigorous evaluation. Furthermore, our research aims to address the limitations of the current evaluation metric, the Collision Rate, which tends to underestimate the collision rate in real-world scenarios. We introduce a new metric, termed Worst-Case Collision Rate. This metric is indicative of the frequency at which the model makes highly risky decisions. Code and information about the dataset is available at https://github.com/ktnguyen17/nuScenesComplex.