ATOM: Automated Black-Box Testing of Multi-Label Image Classification Systems

被引:2
作者
Hu, Shengyou [1 ,2 ]
Wu, Huayao [1 ,2 ]
Wang, Peng [1 ,2 ]
Chang, Jing [3 ]
Tu, Yongjun [3 ]
Jiang, Xiu [3 ]
Niu, Xintao [1 ,2 ]
Nie, Changhai [1 ,2 ]
机构
[1] Nanjing Univ, State Key Lab Novel Software Technol, Nanjing, Peoples R China
[2] Nanjing Univ, Dept Comp Sci & Technol, Nanjing, Peoples R China
[3] Guangdong OPPO Mobile Telecommun Corp Ltd, Guangzhou, Guangdong, Peoples R China
来源
2023 38TH IEEE/ACM INTERNATIONAL CONFERENCE ON AUTOMATED SOFTWARE ENGINEERING, ASE | 2023年
基金
中国国家自然科学基金;
关键词
Multi-label Image Classification Testing; Black-box Testing; Metamorphic Testing;
D O I
10.1109/ASE56229.2023.00156
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
Multi-label Image Classification Systems (MICSs) developed based on Deep Neural Networks (DNNs) are extensively used in people's daily life. Currently, although there are a variety of approaches to test DNN-based systems, they typically rely on the internals of DNNs to design test cases, and do not take the core specification of MICS (i.e., correctly recognizing multiple objects in a given image) into account. In this paper, we propose ATOM, an automated and systematic black-box testing framework for testing MICS. Specifically, ATOM exploits the label combination as the testing adequacy criteria, hoping to systematically examine the impact of correlations between a fixed number of labels on the classification ability of MICS. Then, ATOM leverages image search engine and natural language processing to find test images that are not only common to the real-world, but also relevant to target label combinations. Finally, ATOM combines metamorphic testing and label information to realize test oracle identification, based on which the ability of MICS in classifying different label combinations is evaluated. To evaluate the effectiveness of ATOM, we have performed experiments on two popular datasets of MICS, VOC and COCO (each with five state-of-the-art DNN models), and one real-world photo tagging application from our industrial partner. The experimental results reveal that the performance of current DNN-based MICSs remains less satisfactory even in recognizing correlations between only two labels, as ATOM triggers a total number of 6,049 such label combination related errors for all MICSs studied. In particular, ATOM reports 587 error-revealing images for the industrial MICS, in which 92% of them are confirmed by the developers.
引用
收藏
页码:230 / 242
页数:13
相关论文
共 75 条
[31]   The Emerging Trends of Multi-Label Learning [J].
Liu, Weiwei ;
Wang, Haobo ;
Shen, Xiaobo ;
Tsang, Ivor W. .
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 2022, 44 (11) :7955-7974
[32]   DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems [J].
Ma, Lei ;
Juefei-Xu, Felix ;
Zhang, Fuyuan ;
Sun, Jiyuan ;
Xue, Minhui ;
Li, Bo ;
Chen, Chunyang ;
Su, Ting ;
Li, Li ;
Liu, Yang ;
Zhao, Jianjun ;
Wang, Yadong .
PROCEEDINGS OF THE 2018 33RD IEEE/ACM INTERNATIONAL CONFERENCE ON AUTOMTED SOFTWARE ENGINEERING (ASE' 18), 2018, :120-131
[33]  
Ma L, 2019, 2019 IEEE 26TH INTERNATIONAL CONFERENCE ON SOFTWARE ANALYSIS, EVOLUTION AND REENGINEERING (SANER), P614, DOI [10.1109/saner.2019.8668044, 10.1109/SANER.2019.8668044]
[34]  
MCAR, about us
[35]   Interrater reliability: the kappa statistic [J].
McHugh, Mary L. .
BIOCHEMIA MEDICA, 2012, 22 (03) :276-282
[36]  
MLD, about us
[37]  
MLGCN, about us
[38]  
MSRN, about us
[39]  
Murphy C, 2007, P 2 INT WORKSH RAND, P38, DOI DOI 10.1145/1292414.1292425
[40]  
Murphy C, 2007, SEKE, V2007, P167