Metamorphic Testing Harness for the Baidu Apollo Perception-Camera Module

被引：7

作者：

Zhang, Yifan ^{[1
]}

Towey, Dave ^{[1
]}

Pike, Matthew ^{[1
]}

Han, Jia Cheng ^{[2
]}

Zhou, George ^{[3
]}

Yin, Chenghao ^{[4
]}

Wang, Qian ^{[4
]}

Xie, Chen ^{[4
]}

机构：

[1] Univ Nottingham Ningbo China, Sch Comp Sci, Ningbo 315100, Zhejiang, Peoples R China

[2] Univ Wollongong, Sch Comp & Informat Technol, Inst Cybersecur & Cryptol, Wollongong, NSW 2522, Australia

[3] NIO, Dept Digital Safety & Cybersecur, Shanghai, Peoples R China

[4] NIO, AD Simulat Dept, Shanghai 201804, Peoples R China

来源：

2023 IEEE/ACM 8TH INTERNATIONAL WORKSHOP ON METAMORPHIC TESTING, MET | 2023年

关键词：

Autonomous driving system (ADS); metamorphic testing (MT); metamorphic relation (MR); oracle problem; automated testing;

D O I：

10.1109/MET59151.2023.00009

中图分类号：

TP31 [计算机软件];

学科分类号：

081202 ; 0835 ;

摘要：

As the complexity of autonomous driving systems (ADSs) increases, the question of how to organize testing in an efficient manner has become a serious issue. This study investigates the potential for metamorphic testing (MT) to evaluate the perception-camera module of an open-source autonomous driving system (ADS), namely Baidu's Apollo ADS. The experiments revealed inconsistent obstacle identification results when increasing the brightness of a specific region of the driving scenarios, both in individual and sequential frames, demonstrating the ability of MT to address the oracle problem when testing the perception module of ADSs. Furthermore, this paper presents an MT harness to facilitate ADS testing, which would increase efficiency and help testers to better organize the testing procedure. We also present an industry case study to demonstrate its use in actual production phases.

引用

页码：9 / 16

页数：8

共 23 条

[11]

Geiger A, 2012, PROC CVPR IEEE, P3354, DOI 10.1109/CVPR.2012.6248074

[12]

Han J. C., 2020, P IEEE ACM 42 INT C, P380, DOI [DOI 10.1145/3387940.3392252, 10.1145/3387940.3392252]

[13] Difference Grouping and Test Suite Evaluation: Lessons from Automated Differential Testing for Adobe Analytics [J].

Jarman, Darryl ;

Hunt, Scott ;

Berry, Jeffrey ;

Towey, Dave .

2020 IEEE/ACM 15TH INTERNATIONAL CONFERENCE ON AUTOMATION OF SOFTWARE TEST, AST, 2020, :103-104

[14]

Muller M.U., ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci. V-1-2020, P33, DOI [10.5194/isprs-annals-V-1-2020-, DOI 10.5194/ISPRS-ANNALS-V-1-2020-33-2020, 10.5194/isprs-annals-v-1-2020-33-2020]

[15] Performance of Open Autonomous Vehicle Platforms: Autoware and Apollo [J].

Raju, Vysyaraju Manikanta ;

Gupta, Vrinda ;

Lomate, Shailesh .

2019 IEEE 5TH INTERNATIONAL CONFERENCE FOR CONVERGENCE IN TECHNOLOGY (I2CT), 2019,

[16]

Redmon J, 2016, Arxiv, DOI arXiv:1612.08242

[17] A Survey on Metamorphic Testing [J].

Segura, Sergio ;

Fraser, Gordon ;

Sanchez, Ana B. ;

Ruiz-Cortes, Antonio .

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, 2016, 42 (09) :805-824

[18] To what extent do DNN-based image classification models make unreliable inferences? [J].

Tian, Yongqiang ;

Ma, Shiqing ;

Wen, Ming ;

Liu, Yepang ;

Cheung, Shing-Chi ;

Zhang, Xiangyu .

EMPIRICAL SOFTWARE ENGINEERING, 2021, 26 (05)

[19] DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars [J].

Tian, Yuchi ;

Pei, Kexin ;

Jana, Suman ;

Ray, Baishakhi .

PROCEEDINGS 2018 IEEE/ACM 40TH INTERNATIONAL CONFERENCE ON SOFTWARE ENGINEERING (ICSE), 2018, :303-314

[20] Image quality assessment: From error visibility to structural similarity [J].

Wang, Z ;

Bovik, AC ;

Sheikh, HR ;

Simoncelli, EP .

IEEE TRANSACTIONS ON IMAGE PROCESSING, 2004, 13 (04) :600-612

← 1 2 3 →