A Survey of Coverage-Based Testing Tools

Test coverage is sometimes used to measure how thoroughly software is tested and developers and vendors sometimes use it to indicate their confidence in the readiness of their software. This survey studies and compares 17 coverage-based testing tools primarily focusing on, but not restricted to, coverage measurement. We also survey features such as program prioritization for testing, assistance in debugging, automatic generation of test cases and customization of test reports. Such features make tools more useful and practical, especially for large-scale, commercial software applications. Our initial motivations were both to understand the available test coverage tools and to compare them to a tool that we have developed, called eXVantage (a tool suite that includes code coverage testing, debugging, performance profiling and reporting). Our study shows that each tool has some unique features tailored to its application domains. The readers may use this study to help pick the right coverage testing tools for their needs and environment. This paper is also valuable to those who are new to the practice and the art of software coverage testing, as well as those who want to understand the gap between industry and academia.