The WASS Project which Veracode contributed data to shows some nice benefits to White box (static) over Black box (dynamic) for many serious vulnerability categories. White box overall detects a higher prevalence of many categories which we can extrapolate to having lower FN rates. Now the sample set of apps is not the same so this can only be used as a trend. Static is better than dynamic in 5 out of 7 categories: credential/session prediction, SQL Injection, Path Traversal, Insufficient Authorization, OS Commandeering. In one category, insufficient authorization, dynamic is better and in one category, brute force attack, my gut feel is this is within the margin of error given the different app samples.

I consider credential/session prediction flaws detected by white box to be typically hard to exploit even though it is a real flaw. White box (static) analysis reports this whenever non-cryptographically strong random number generators are used to generate session identifiers or resource IDs. Usually this means standard rand() is used. The SQL injection, path traversal, and OS commandeering are probably found better by static because these are a good sweet spot for static with its 100% code coverage. All that is required is good data flow modeling from web request to tainted function. In this case, database query, file I/O, or system/process calls. Black box not finding as much is likely do to much less coverage of code paths in the application.

Percent of vulnerabilities out of total number of vulnerabilities (% Vulns BlackBox & WhiteBox)

If we consider the prevalence of high risk level vulnerabilities in detailed web application analysis (P. 9) we’ll see that the most widespread is Credential/Session Prediction errors. SQL Injection, Path Traversal and implementation and configuration errors in authentication and authorization systems are also widespread.

Veracode Security Solutions
Veracode Security Threat Guides

Written by:

About Chris Wysopal

Chris Wysopal, co-founder and CTO of Veracode, is recognized as an expert and a well-known speaker in the information security field. He has given keynotes at computer security events and has testified on Capitol Hill on the subjects of government computer security and how vulnerabilities are discovered in software. His opinions on Internet security are highly sought after and most major print and media outlets have featured stories on Mr. Wysopal and his work. At Veracode, Mr. Wysopal is responsible for the security analysis capabilities of Veracode technology.

Comments (2)

Dave Hull | October 21, 2009 12:28 pm

I read through the WASS data yesterday and found it matched nicely with my experience. I've been working in app sec for a few years now, using the full gamut of threat modeling, static code analysis and manual testing. I see white box testing progressing along Stu Feldman's maturity model:

1. You have a good idea.
2. You can make it work.
3. You convince a gullible friend to try it.
4. People stop asking why you're doing it.
5. People start asking others why they aren't doing it.

In the not too distant future, the companies that aren't doing white box testing will be the outliers.

Black box testing has too many unknowns and they are mostly unknown unknowns (thank you Rumsfeld).

Juan Gama | November 16, 2009 2:12 pm

You are right about the coverage, WhiteBox has a better results over BlackBox, but the problem that I've been seeing is the cost, maybe is better to have a WhiteBox test but it is also more expensive, maybe this is the main retractor when companies do not want a internal security team and instead of it they go to external companies to perform a security test.

Besides BlackBox tests are way cooler than WhiteBox tests :P

Please Post Your Comments & Reviews

Your email address will not be published. Required fields are marked *

Love to learn about Application Security?

Get all the latest news, tips and articles delivered right to your inbox.