Practical black-box adversarial attack on open-set recognition: Towards robust autonomous driving

As an important method of image classification, Open-Set Recognition (OSR) has been gradually deployed in autonomous driving systems (ADSs) for detecting the surrounding environment with unknown objects. To date, many researchers have demonstrated that the existing OSR classifiers are heavily threatened by adversarial input images. Nevertheless, most existing attack approaches are based on white-box attacks, assuming that information of the target OSR model is known by the attackers. Hence, these attack models cannot effectively attack ADSs that keep models and data confidential. To facilitate the design of future generations of robust OSR classifiers for safer ADSs, we introduce a practical black-box adversarial attack. First, we simulate a real-world open-set environment by reasonable dataset division. Second, we train a substitute model, in which, to improve the transferability of the adversarial data, we combine dynamic convolution into the substitute model. Finally, we use the substitute model to generate adversarial data to attack the target model. To the best of the authors' knowledge, the proposed attack model is the first to utilize dynamic convolution to improve the transferability of adversarial data. To evaluate the proposed attack model, we conduct extensive experiments on four publicly available datasets. The numerical results show that, compared to the white-box attack approaches, the proposed black-box attack approach has a similar attack capability. Specifically, using the German Traffic Sign Recognition Benchmark dataset, our model can decrease the classification accuracy of known classes from 99.8% to 9.81% and can decrease the AUC of detecting unknown classes from 97.7% to 48.8%.