Evaluating software testing techniques: A systematic mapping study

Software testing techniques are crucial for detecting faults in software and reducing the risk of using it. As such, it is important that we have a good understanding of how to evaluate these techniques for their efficiency, scalability, applicability, and effectiveness at finding faults. This article enhances our understanding of software testing technique evaluations by providing an overview of the state of the art in research and structuring the field to assist researchers in locating types of evaluations they are interested in. To do so a systematic mapping study is performed. Three hundred and sixty-five primary studies are systematically collected from the field and each mapped into categories based on numerous classification schemes. This reveals the distribution of research by each category and identifies where there are research gaps. It also results in a mapping from each combination of categories to actual papers belonging to them; allowing researchers to very quickly locate all of the testing technique evaluation research with properties they are interested in. Further classifications are performed on case study and experiment evaluations in order to assess the relative quality of these evaluations. The distribution of research by various category combinations is presented along with a large table mapping each category combination to the papers belonging to them. We find a majority of evaluations are empirical evaluations in the form of case studies and experiments, most of them are of low quality based on proper methodology guidelines, and relatively few papers in the field discuss how testing techniques should be evaluated.