DeepWeak: Weak Mutation Testing for Deep Learning Systems

The widespread application of deep learning (DL) makes it crucial to ensure its reliability. Mutation testing has been employed in DL testing to evaluate the quality of test suite. However, the largest problem of DL mutation testing is the high cost of executing numbers of mutants. Weak mutation technology can alleviate this problem by reducing the execution time of mutants in traditional software testing. However, the compared components in traditional software are too trivial to apply weak mutation to DL models directly for that it is impractical for testers to track and monitor massive parameters during execution process. In this paper, we propose a novel weak mutation framework for mutants generated by source-level mutation operators. DeepWeak treats all layers that make up the DL model directly as a set of components of model to replace trivial parameters. And it pays attention to the last convolutioanl layer for that they not only have impacts on prediction results but also are evident for weak analysis. By quantifying contribution of feature maps to the prediction, weight maps will be obtained on the basis of their weights. Finally, the judgements on whether mutants have been killed will be reached by comparing the maps. To evaluate the applicability and effectiveness of our approach, we conduct experiments on three widely used datasets and four deep learning models using three metrics. Experimental results show that DeepWeak is effective at alleviating costs problem, reducing runtime by 11.21% to 18.21% compared with the DL mutation testing with little accuracy loss.