Broken Logic: Avoiding the Test Site Fallacy

Web security scanners are one tool in the arsenal of any organization that takes security seriously. The ability of automation to rapidly test and verify that an application meets a reasonable standard of security is a key advantage. While manual testing can never be completely removed from the process, automated tools are critical in reducing the amount of time spent on repetitive tasks. In some cases applications are so large that it is not possible for a single human to cover even a small portion of the application’s functionality. Dynamic Application Security Testing (DAST) has become an integral part of the Secure Development Lifecycle of most organizations. While each scanner has its own strengths and weaknesses, they all are designed to achieve common goals. DAST tool vendors demonstrate the effectiveness of their tools by allowing prospective customers to scan sites so they can see how the scanner works and the type of information it reports. Some scanners offer trial versions restricted to scanning a single site that the vendor themselves created and operate. In the end, we should not gauge the abilities and effectiveness of a particular scanner by only looking at the results from scanning these public test sites. Overall, problems with these sites ended up falling into five key areas of concern:

  • Sites are closed-source
  • Sites have missing and incomplete technology or vulnerability coverage
  • Sites are broken
  • Sites contain unrealistic or fake vulnerabilities
  • Sites contain unrealistic form validation or checks

Ultimately, these test sites were built to make the vendor's product look good. 

Isaac Dawson is a Senior Security Research at Veracode, where he researches attacks and methods for creating a next generation application security scanner. This cutting edge research has led to advancements in Veracode’s automated web scanner’s ability to identify and exploit security issues. He has developed numerous internal systems containing thousands of variations of web vulnerabilities for use in ensuring the greatest amount of scanner coverage.