[UPDATE: Since there seems to be some confusion, the "We" in the title of this post is NOT "Veracode". The expression is a generic one intended to illustrate the attitude exhibited by many companies who like to downplay the value and/or effectiveness of technologies that they themselves do not sell. I can't believe I am having to explain this.]

Fair warning, this is a bit of a rant.

Back in my consulting days (early 2000, I'm getting old), we delighted in the fact that our web application penetration testing methodology didn't rely on automated tools. This was completely true; we did everything manually, and we were among the best in the industry. Many so-called security consultants of the day would run a commercial web scanner and repackage the results as a high dollar "penetration test" -- what a ripoff!

What we didn't acknowledge to our customers is that those web scanners, even in their immature state, were probably capable of detecting some of the low hanging fruit that we didn't want to spend our time looking for. Oh, we'd find a few "representative examples" of XSS and SQL injection, but then we'd get bored and move on to the more interesting and complex attack vectors. In our naivete, we figured developers would be inspired to revisit their entire input validation and/or output encoding practices, as opposed to just fixing the proof-of-concept examples we found.

Meanwhile, the commercial web scanner vendors were always downplaying the value of manual testing! "Why would you want to pay for an expensive penetration test when you can just run this less expensive tool and find the same vulnerabilities?" They'd gloss over all the technical challenges of automated web scanning and conveniently forget to mention how it was impossible for them to find authorization issues, cryptographic weaknesses, business logic flaws, and so on.

What's my point?

Using multiple testing methodologies is crucial. Sure, there may be some overlap, but ultimately they are complementary to one another. That's why at Veracode, we've never positioned automated static analysis (SAST) as a complete solution. That's why we integrated both automated web scanning (DAST) and manual penetration testing into our service offerings less than a year after launching the company, even though SAST is our patented bread-and-butter technology. This meant we could always be completely honest about the strengths and weaknesses of each technique. I've had a slide titled "There Is No Silver Bullet" in my corporate slide deck since the very beginning.

Our silver bullet is better than yours

Meanwhile, it's been amusing to watch other companies -- who only had a single offering -- having to espouse the tactic of downplaying any testing approach that wasn't in their service portfolio.

I'm only picking on these guys because they're visible, well-respected practitioners in the application security space. Of course Brian knows source code scanning is an incomplete solution, and now that Fortify and WebInspect are part of the same parent company, I suspect he's adjusted his message. I'm certain Jeremiah knows there's value in writing secure code during the SDLC, which is why WhiteHat is now trying to get into the SAST market by acquiring some technology.

And I'm pretty sure Dave Maynor knows automation does provide real value. How else can a big company -- spooked by all the recent breaches -- quickly hunt for SQL injection vulnerabilities across 5,000 websites without the benefit of automation? How does one look for issues in the 150 third-party libraries you use, where only the binary is available? Do you hire Mark Dowd to spend a month looking at each one?

Building trust

We all know a few sales reps that jump from one company to another, changing their pitch as they go no matter how much it conflicts with things they've said in the past. First a service-based approach is best, but suddenly an on-premise tool is better. Source code scanning used to be pointless, but now it's the best thing since sliced bread! It's no surprise these guys don't experience more success -- they lack credibility. The most successful account reps I've seen are the ones who build trust with their customers over time by being honest about what they are selling, even when hopping from one company to the next.

Look, it's no big secret why people talk up their own stuff and imply everything else stinks. It's part of the sales and marketing machine and by no means is it unique to the security industry. Even so, can't we make an effort -- as practitioners -- to cut back on the rhetoric a little bit and be more honest with our customers? Customers look to us as experts to help them build their security programs, and what do we do? We oversell them on an approach that has huge gaps we pretend don't exist. If you're really looking out for your customers, start being more honest, and stop handing out kool-aid.

Here's another approach: Instead of outright dismissing an effective technology or methodology just because you don’t sell it, sometimes it's worth thinking about partnering, or even building something better. That's why at Veracode we designed our service platform around the idea of technology integration. There is no silver bullet and there never will be.

Veracode Security Solutions
Security Alternatives
Security Threat Guides

About Chris Eng

Chris Eng, vice president of research, is responsible for integrating security expertise into Veracode’s technology. In addition to helping define and prioritize the security feature set of the Veracode service, he consults frequently with customers to discuss and advance their application security initiatives. With over 15 years of experience in application security, Chris brings a wealth of practical expertise to Veracode.

Comments (5)

Andre Gironda | July 6, 2011 11:47 am

Full automation, as seen in the way that these products and services are bought and sold today (especially Veracode, WhiteHat Security, Fortify, etc) are doomed to fail. I agree with ErrataSec -- we need to change.

However, ErrataSec didn't say that they don't use automation. Everyone with half a brain uses automation, but they use it sparingly and only when appropriate. Do you think that a prototyping language such as Python, Ruby, bash, Lua, Haskell, et al is more capable at creating a sustainable debugging-wrapper around any target application or environment? I think that concurrent programming in these prototyping dynamic langauges can easily scale to 500,000 hosts or 15,000 third-party libraries. A domain-specific language such as O2 provides even faster and more efficient prototyping for these sorts of efforts.

What some web application security scanners provide is the ability to perform a crawl-only scan and export the hosts, URIs, parameters, form data (HTML, Ajax, Flash, et al), XML/JSON/etc data, and other "interesting" data to a set of easily consumed XML files. A prototyping or DSL can consume this XML and provide the necessary runtime automation (DAST) capabilities.

The same is true of SAST. Cigital ESP is the only platform that will locate missing sinks and instinctively "do something about them". However, in the case of missing sources due to one or multiple levels of code indirection (very common in MVC frameworks, especially ones that lean on dependency injection, Plugin, or Command Patterns, etc), the sources will be missing. These will need to be traced and added to the understanding of the SAST framework involved. However, this task doesn't need to be done completely manually. It just involves a bit of human involvement. That's ok though because the application developers want this stuff anyways. They also need a mapping of sources-to-sinks and sinks-to-sources with the ability to trace in either direction.

The cottage industry leans on specific tools to accomplish this. For runtime, they use Burp Suite Professional, along with the IBurpExtender interface -- usually via the JRuby DSL, Buby, or perhaps Burp Python or the native Java interface. For static analysis, the use of IBM Appscan Source Edition is preferred exactly for the same reasons that Cigital ESP includes it. However, I often see combined use of Checkmarx CxSuite because of its extensibility and ability to work on semantically correct (instead of buildable, or built) source code.

Also -- the appsec product/service industry's "full automation" model fails for an additional reason: Grinding through a list of apps as assets (especially "one-at-a-time" or with "one/many-size(s)-fits-all" models) is inefficient and unsustainable. Penetration-testers need to look holistically at the metastructure and infostructure in addition to the social capital and attack it all as a single piece as real-world attackers would (or at least model the threats as reality dictates). Instead of remediation guidance focused on configuration file changes or code changes (especially large, complex architectural ones) -- we instead need to build Technical Intelligence (TECHINT) that crushes our opponents' weapons (and frustrates them in the process) in fancy ways (i.e. it should be something that we can laugh about as they try and fail to breach us). We need to deceive our adversaries by not only "knowing our enemy" but also working against their technologies on a case-by-case basis.

Billy Hoffman | July 6, 2011 12:53 pm

Great post Chris. It's amazing to me that vendors can talk about defense in depth while simultaneously bad mouthing other approaches to the problem. We need less marketing dollars spent attacking each other and more spent on getting security products, really any at all, into the hands of web app devs.

CEng | July 6, 2011 1:14 pm

@Andre: You're really reinforcing my point. Automation alone is not a complete solution, nor is highly-customized manual tooling. The former isn't specialized enough; the latter has no hope of scaling. There are distinct advantages and disadvantages to SAST, DAST, penetration testing, customized tools, and so on. Nothing is comprehensive.

I disagree with your opinion that apps shouldn't be inventoried and viewed as assets -- you can't begin to understand your exposure until you know what you have to begin with. It's difficult, but essential. Technical Intelligence, whatever that entails, sounds nice in theory but clearly needs to become less nebulous in order to be useful.

Andre Gironda | July 6, 2011 2:47 pm

@ Chris:

I didn't say "don't inventory apps/assets". I said that the Enterprise risk management model that uses assets in this way is failing. Perhaps we only need more time and energy to demonstrate success. But perhaps also that there are things we can do in the meantime to improve on existing models for appsec risk management?

I believe that Technical Intelligence, along with Threat Intelligence, will become less nebulous than Enterprise risk management for application security, data security, and general information security controls because it is rooted in offensive security. It may certainly be a theory (as applied to information security) at this time, but it has been practiced in warfare (and therefore kinetic security) for hundreds of centuries. TECHINT has shown past success in kinetic security when paired against the similar problems we are currently facing in cyber security.

What Veracode, Fortify On-Demand, and WhiteHat Security provide does not equate to what Attack Research, Immunity Security, or Errata Security provide. Yes, all 3 of the hybrid, SAST, and DAST services you mentioned do have humans involved in the process. My list of 3 go beyond the capabilities provided in hybrid, SAST, and DAST people, process, and tools. They perform OSINT, TECHINT, SIGINT, and other intelligence services related to offensive security -- matching adversarial capabilities to information security controls and vice-versa.

Our argument is really about how and where to focus attention and specific types of resources.

Marisa | January 22, 2014 1:26 pm

Chris, your disclaimer is in the wrong place. People only read the headlines!

Great rant. I enjoyed it. Write more often! :)

Please Post Your Comments & Reviews

Your email address will not be published. Required fields are marked *

Love to learn about Application Security?

Get all the latest news, tips and articles delivered right to your inbox.