A Case Study of Concolic Testing Tools and Their Limitations

Automatic testing, in particular test input generation, has become increasingly popular in the research community over the past ten years. In this paper, we conduct a survey on existing concolic testing tools, discussing their strengths and limitations, and environments in which they can be applied. We also conduct a case study to determine the prevalence of the identified limitations in six large software systems (four from open-source and two from ABB), as well as the effectiveness and scalability of the publicly available tools. The results show that pointers and native calls are the most prevalent limitations, preventing tools from generating high branch coverage test cases, and variables of float type are the least prevalent. The scalability of the publically available tools is also a limitation for industrial use, due to the large overhead of creating a test harness. Finally, we propose suggestions on how practitioners can use these tools and how researchers can improve concolic testing.