DLFuzz: Differential Fuzzing Testing of Deep Learning Systems

被引:183
作者
Guo, Jianmin [1 ,3 ]
Jiang, Yu [1 ]
Zhao, Yue [1 ]
Chen, Quan [2 ]
Sun, Jiaguang [1 ]
机构
[1] Tsinghua Univ, Sch Software, Beijing, Peoples R China
[2] Shanghai Jiao Tong Univ, Dept Comp Sci & Engn, Shanghai, Peoples R China
[3] Beijing Natl Res Ctr Informat Sci & Technol BNRis, Minist Educ, Key Lab Informat Syst Secur, Beijing, Peoples R China
来源
ESEC/FSE'18: PROCEEDINGS OF THE 2018 26TH ACM JOINT MEETING ON EUROPEAN SOFTWARE ENGINEERING CONFERENCE AND SYMPOSIUM ON THE FOUNDATIONS OF SOFTWARE ENGINEERING | 2018年
关键词
Fuzzing Testing; Deep Learning; Neuron Coverage;
D O I
10.1145/3236024.3264835
中图分类号
TP31 [计算机软件];
学科分类号
081202 ; 0835 ;
摘要
Deep learning (DL) systems are increasingly applied to safety critical domains such as autonomous driving cars. It is of significant importance to ensure the reliability and robustness of DL systems. Existing testing methodologies always fail to include rare inputs in the testing dataset and exhibit low neuron coverage. In this paper, we propose DLFuzz, the first differential fuzzing testing framework to guide DL systems exposing incorrect behaviors. DLFuzz keeps minutely mutating the input to maximize the neuron coverage and the prediction difference between the original input and the mutated input, without manual labeling effort or cross-referencing oracles from other DL systems with the same functionality. We present empirical evaluations on two well-known datasets to demonstrate its efficiency. Compared with DeepXplore, the state-of-the-art DL whitebox testing framework, DLFuzz does not require extra efforts to find similar functional DL systems for cross-referencing check, but could generate 338.59% more adversarial inputs with 89.82% smaller perturbations, averagely obtain 2.86% higher neuron coverage, and save 20.11% time consumption.
引用
收藏
页码:739 / 743
页数:5
相关论文
共 21 条
[1]  
[Anonymous], 2018, ARXIV180700182
[2]  
[Anonymous], 2014, Towards deep neural network architectures robust to adversarial examples
[3]  
Bojarski Mariusz, 2016, arXiv
[4]  
Deng J, 2009, PROC CVPR IEEE, P248, DOI 10.1109/CVPRW.2009.5206848
[5]   Deep Residual Learning for Image Recognition [J].
He, Kaiming ;
Zhang, Xiangyu ;
Ren, Shaoqing ;
Sun, Jian .
2016 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 2016, :770-778
[6]   Connectomic reconstruction of the inner plexiform layer in the mouse retina [J].
Helmstaedter, Moritz ;
Briggman, Kevin L. ;
Turaga, Srinivas C. ;
Jain, Viren ;
Seung, H. Sebastian ;
Denk, Winfried .
NATURE, 2013, 500 (7461) :168-+
[7]   Safety Verification of Deep Neural Networks [J].
Huang, Xiaowei ;
Kwiatkowska, Marta ;
Wang, Sen ;
Wu, Min .
COMPUTER AIDED VERIFICATION, CAV 2017, PT I, 2017, 10426 :3-29
[8]   ImageNet Classification with Deep Convolutional Neural Networks [J].
Krizhevsky, Alex ;
Sutskever, Ilya ;
Hinton, Geoffrey E. .
COMMUNICATIONS OF THE ACM, 2017, 60 (06) :84-90
[9]   Gradient-based learning applied to document recognition [J].
Lecun, Y ;
Bottou, L ;
Bengio, Y ;
Haffner, P .
PROCEEDINGS OF THE IEEE, 1998, 86 (11) :2278-2324
[10]  
LeCun Y., 2010, The mnist database of handwritten digits, P2