One Technique is Not Enough: A Comparison of Vulnerability Discovery Techniques

Security vulnerabilities discovered later in the development cycle are more expensive to fix than those discovered early. Therefore, software developers should strive to discover vulnerabilities as early as possible. Unfortunately, the large size of code bases and lack of developer expertise can make discovering software vulnerabilities difficult. To ease this difficulty, many different types of techniques have been devised to aid developers in vulnerability discovery. The goal of this research is to improve vulnerability detection by comparing the effectiveness of vulnerability discovery techniques and to provide specific recommendations to improve vulnerability discovery with these techniques. We conducted a case study on two electronic health record systems to compare four discovery techniques: systematic and exploratory manual penetration testing; static analysis; and automated penetration testing. In our case study, we found empirical evidence that no single technique discovered every type of vulnerability. We discovered almost no individual vulnerabilities with multiple discovery techniques. We also found that systematic manual penetration testing found the most design flaws, while static analysis found the most implementation bugs. Finally, we found the most effective vulnerability discovery technique in terms of vulnerabilities discovered per hour was automated penetration testing. These results suggest that if one has limited time to preform vulnerability discovery one should conduct automated penetration testing to discover implementation bugs and systematic manual penetration testing to discover design flaws.