Thanks for weighing in. I can certainly understand and appreciate the effort involved in pulling together this amount of data and to that extent I believe this project is a step in the right direction. My motivation behind writing this post was not to malign the project itself, but rather to provide some much-needed context and to help others interpret the data.

I really don’t think your site points out the limitations and inaccuracies of the data set very clearly. For example, it states that “statistical biases will be lessened as more entities contribute to the initiative,” which really still doesn’t address the underlying problem. That statement implies that the statistical deficiencies stem from not having a large or diverse enough sample set. However, the size of the data set and the number of vendors is not the issue, it’s the data itself. Case in point, I can’t think of any upside to commingling raw scan data with validated results. It just doesn’t make sense — it pollutes and obscures the data that is meaningful to most readers.

It’s a given that the stats shouldn’t be “taken as gospel truth that all websites will have a like distribution of vulnerabilities,” and I don’t think anyone would reasonably expect that. My point is that these numbers aren’t even in the ballpark.

I appreciate your critical review of the WASC Web Application Security Statistics. You are absolutely correct that the statistics ‘reflect the capabilities and limitations of [the] scanners [used]‘. You are also correct that these numbers include a sampling bias. We certainly did not intend for these statistics to be taken as gospel truth that all websites will have a like distribution of vulnerabilities. We do however strongly believe that web application vulnerabilities are a growing problem that will not go away unless we do something about them. In describing the purpose of the project we did our very best to explain that this is an emerging initiative with limitations, not a flawless piece of scientific data.

Having competing firms band together to share data to provide to the community is a positive step forward. We have reasonable stats on COTS software but very little to provide insight into what we’re seeing in custom web applications. The WASC Web Application Security Statistics project is a first step, not the end of the journey. I hope that other vendors will join the initiative going forward to both provide additional data to remove statistical biases and to continue to critique the initiative as you have done here. I know that Veracode conducts assessments of web applications and hope that Veracode will consider joining this initiative to ensure that we are able to provide increasingly accurate statistics.

Regards,

Michael Sutton
WASC Web Application Security Statistics
Project Leader