The recurring theme in her manifesto is the notion that certain software suppliers are “too big to test”. It’s fine for the little guys, but not the 800-pound gorillas. Instead, software purchasers should blindly trust companies with security teams and assurance processes to produce secure code. If only it were that simple. In fact, according to our semi-annual State of Software Security Report, there’s negligible variation in security quality across software suppliers regardless of company size.

We’re both flattered and amused that Ms. Davidson believes our company alone “created a market” for testing the software supply chain. On the contrary, the market has created itself. Take a look through the noteworthy breaches from the past 12-24 months; software vulnerabilities have been the culprit in nearly every case. CISOs are waking up to the stark realization that all software — internally or externally produced — introduces risk into their organizations. In this day and age, wise companies harbor a healthy suspicion of their software vendors. Oracle can choose to do security testing in-house, but a company that’s “running their entire business” on Oracle’s software has a right to request unbiased evidence that the testing process is working.

That being said, Oracle is hardly the poster child for security process. Within the security community, they are notorious for shipping insecure products. Their laughable “Unbreakable” marketing campaign was famously debunked by security expert David Litchfield, who uncovered several critical (and easily avoidable) vulnerabilities within a matter of weeks. They’ve also earned a reputation for glacial response times and sloppy patches. No company can be expected to build perfectly secure software, but it’s pretty obvious why external validation is needed to complement in-house process — one need look no further than ZDI for evidence. Even Ms. Davidson’s own example illustrates how an outsourced service provider “HuiMaika’i” detected multiple vulnerabilities that weren’t discovered by Oracle’s internal team.

Perhaps the most shocking admission about Oracle’s security program is their interpretation of the “need to know” principle. Ms. Davidson asserts that she doesn’t need access to bug databases. This is a classic liability avoidance move and one that we’ve witnessed in other organizations as well. Creating barriers to vulnerability information facilitates a culture in which the executive has plausible deniability of critical bugs and can simply look the other way if a ship deadline is looming or if the auditors pay a visit. CISOs should be clamoring for as much data as they can get their hands on, not eschewing it.

Finally, Ms. Davidson seemed offended that a tenured university professor would suggest licensing software developers to create a system of accountability. Ironically, only a few years ago, she sent a letter to top universities pressuring them to incorporate secure coding guidelines such as the SANS coding certification into their curriculums. She told them, “We will start making our purchasing decisions, if you will, based on that.” Apparently, it’s OK for Oracle to flex their muscle when “buying” (i.e. hiring) from universities, but it’s not OK for Oracle’s customers to hold them to similar standards? It certainly sounds like Oracle has been feeling the pressure lately.

There are third-party tests and assessments for perhaps every important purchase in business or in our personal lives. Companies hire law firms and specialists when they make acquisitions. People look to safety and quality tests from trusted sources before they buy everything from baby strollers to cars. You wouldn’t think of buying a home without a home inspection. In each case, the cost of the independent test must be commensurate with the purchase price and the risk. Look at the typical due dilligence around home purchase. It doesn’t always make sense to pay an engineering firm thousands of dollars for a structural analysis, but it does make sense to hire a home inspector for a few hundred dollars, who in a few hours can uncover termites or a leaking roof. These are problems that must be fixed, and because the testing cost is so low it would be negligent not to do it. Most of the major software vendors have participated in third-party testing either as part of their SDLC, to vet code they were acquiring or licensing, or as part of one of their customers’ procurement process.

Veracode has never claimed that binary SAST provides complete software assurance. From the beginning, we have recommended multiple testing methods to detect vulnerabilities that static automation can’t. In fact, it’s impossible to receive our top ratings without a clean bill of health from a manual penetration test. Each layer of testing, while imperfect on its own, uncovers problems that must be corrected.

Outsourcing is not a dirty word. Many companies outsource development for entire products or components of them. Companies also outsource testing and training. The multi-billion dollar IV&V market grew out of this need — it’s simply good business. The goal is shipping secure code, not making a feel-good proclamation that your team can handle a modern development challenge with no outside help. While Oracle can be proud that they have tamed a source code tool and lived to tell the tale, other companies are securing their code faster and cheaper with the help of outsourcing. Even Veracode customers haven’t fully outsourced security; many of them have in-house security expertise and are just employing a service to make their security processes more robust. They are still full participants in the process, making decisions around how/when to remediate, how much to invest, etc. Veracode acts as an application security partner, providing customers valuable intelligence gleaned from the software ecosystem. Just as Google gets smarter with every search that it does, Veracode gets smarter with every scan we do.

At least we can rest easy knowing that Oracle would never hire lobbyists to promote an agenda. That’s a relief!

Veracode Security Solutions

Security Alternatives

Security Threat Guides

Fair warning, this is a bit of a rant.

Back in my consulting days (early 2000, I’m getting old), we delighted in the fact that our web application penetration testing methodology didn’t rely on automated tools. This was completely true; we did everything manually, and we were among the best in the industry. Many so-called security consultants of the day would run a commercial web scanner and repackage the results as a high dollar “penetration test” — what a ripoff!

What we didn’t acknowledge to our customers is that those web scanners, even in their immature state, were probably capable of detecting some of the low hanging fruit that we didn’t want to spend our time looking for. Oh, we’d find a few “representative examples” of XSS and SQL injection, but then we’d get bored and move on to the more interesting and complex attack vectors. In our naivete, we figured developers would be inspired to revisit their entire input validation and/or output encoding practices, as opposed to just fixing the proof-of-concept examples we found.

Meanwhile, the commercial web scanner vendors were always downplaying the value of manual testing! “Why would you want to pay for an expensive penetration test when you can just run this less expensive tool and find the same vulnerabilities?” They’d gloss over all the technical challenges of automated web scanning and conveniently forget to mention how it was impossible for them to find authorization issues, cryptographic weaknesses, business logic flaws, and so on.

What’s my point?

Using multiple testing methodologies is crucial. Sure, there may be some overlap, but ultimately they are complementary to one another. That’s why at Veracode, we’ve never positioned automated static analysis (SAST) as a complete solution. That’s why we integrated both automated web scanning (DAST) and manual penetration testing into our service offerings less than a year after launching the company, even though SAST is our patented bread-and-butter technology. This meant we could always be completely honest about the strengths and weaknesses of each technique. I’ve had a slide titled “There Is No Silver Bullet” in my corporate slide deck since the very beginning.

Our silver bullet is better than yours

Meanwhile, it’s been amusing to watch other companies — who only had a single offering — having to espouse the tactic of downplaying any testing approach that wasn’t in their service portfolio.

• Over at Fortify, Brian Chess famously predicted that 2009 would mark the end of penetration testing.
• Over at WhiteHat, Jeremiah Grossman often downplays the value of writing secure code and testing code quality.
• Even as recently as last week, we have Errata Security (a consultancy) claiming that automated tools are useless and doomed to fail. Welcome back to 1999.

I’m only picking on these guys because they’re visible, well-respected practitioners in the application security space. Of course Brian knows source code scanning is an incomplete solution, and now that Fortify and WebInspect are part of the same parent company, I suspect he’s adjusted his message. I’m certain Jeremiah knows there’s value in writing secure code during the SDLC, which is why WhiteHat is now trying to get into the SAST market by acquiring some technology.

And I’m pretty sure Dave Maynor knows automation does provide real value. How else can a big company — spooked by all the recent breaches — quickly hunt for SQL injection vulnerabilities across 5,000 websites without the benefit of automation? How does one look for issues in the 150 third-party libraries you use, where only the binary is available? Do you hire Mark Dowd to spend a month looking at each one?

Building trust

We all know a few sales reps that jump from one company to another, changing their pitch as they go no matter how much it conflicts with things they’ve said in the past. First a service-based approach is best, but suddenly an on-premise tool is better. Source code scanning used to be pointless, but now it’s the best thing since sliced bread! It’s no surprise these guys don’t experience more success — they lack credibility. The most successful account reps I’ve seen are the ones who build trust with their customers over time by being honest about what they are selling, even when hopping from one company to the next.

Look, it’s no big secret why people talk up their own stuff and imply everything else stinks. It’s part of the sales and marketing machine and by no means is it unique to the security industry. Even so, can’t we make an effort — as practitioners — to cut back on the rhetoric a little bit and be more honest with our customers? Customers look to us as experts to help them build their security programs, and what do we do? We oversell them on an approach that has huge gaps we pretend don’t exist. If you’re really looking out for your customers, start being more honest, and stop handing out kool-aid.

Here’s another approach: Instead of outright dismissing an effective technology or methodology just because you don’t sell it, sometimes it’s worth thinking about partnering, or even building something better. That’s why at Veracode we designed our service platform around the idea of technology integration. There is no silver bullet and there never will be.

Veracode Security Solutions

Security Alternatives

Security Threat Guides

In the realm of automated web application testing, today’s technologies fall into one of two categories, Static Application Security Testing (SAST) and Dynamic Application Security Testing (DAST). SAST analyzes application binaries or source code, detecting vulnerabilities by identifying insecure code paths without actually executing the program. In contrast, DAST detects vulnerabilities by conducting attacks against a running instance of the application, simulating the behavior of a live attacker. Most enterprises have incorporated at least one SAST or DAST technology; those with mature SDLCs may even use more than one of each.

In the past year or so, industry analysts and product vendors have become enamored with so-called “hybrid analysis” technologies. Hybrid techniques aim to correlate the results of SAST and DAST to dramatically expand dynamic coverage, prioritize the combined set of results, and reduce both false positives and false negatives. This whitepaper will examine each of these claims to give consumers technical insight into whether hybrid technologies can realistically live up to the hype.

Several observations will be described in the following sections:

• Hybrid analysis may expand dynamic coverage, but the lack of application context limits its effectiveness.
• The challenge of reliably generating URL-to-source mappings, coupled with the existence of URL rewriting, undermines the accuracy and usefulness of vulnerability correlation.
• Hybrid analysis does not reduce false positive rates; rather, it lulls users into a false sense of security by suggesting that non-correlated vulnerabilities are false positives.
• Correlation should not be equated with exploitability. Vulnerabilities should be prioritized based on severity and business impact, not based on how many scanners are capable of detecting it.

Download the full whitepaper.

Veracode Security Solutions

Veracode Security Threat Guides

@dinozaizovi: Finding vulnerabilities without exploiting them is like putting on a dress when you have nowhere to go.

This clever analogy spurred a discussion about the importance of proving exploitability as a prerequisite to fixing bugs. While I agree that nothing is more convincing than a working exploit, there will always be a greater volume of bugs discovered than there are vulnerability researchers to write exploits. Don’t get me wrong — as a former penetration tester, I agree that it is fun to write exploits, it just shouldn’t be a gating factor. Putting the burden of proof on the researcher to develop an exploit is not scalable, nor does it help create a development culture that improves software security over the long term.

A related topic, and one that hits closer to home for me, is how software developers deal with the results of static analysis. Static analysis is often misunderstood, particularly by people who have only dealt with dynamic analysis (fuzzing, web scanning, etc.) or penetration testing in the past. Because static analysis detects flaws without actually executing the target application, there’s an increased likelihood of finding “noise” (insignificant flaws) or false positives. On the other hand, static analysis provides broader coverage, often detecting flaws in complex code paths that a web scan or human tester would be unlikely to find. So there’s your trade-off.

Here’s a conversation I have all too frequently, paraphrased:

Now imagine that conversation stretching out to 30 minutes or more. They could’ve fixed a half-dozen flaws already. And it’s not limited to SQL injection. For example, consider cross-site scripting (XSS):

Buffer overflows:

By now you get the picture. Many developers want proof, to the extent that they’ll sacrifice efficiency to get it. If we are to improve software over the long haul, developers must learn to recognize situations where it takes less time to patch a bug than to argue about its exploitability. On a more positive note, from someone who talks to static analysis customers on a daily basis, the tide is starting to turn in the right direction. But it is still an uphill battle.

Veracode Security Solutions

Veracode Security Threat Guides

Scrawlr quickly came under fire on the Web Security mailing list for having some pretty major limitations. Billy Hoffman et al have been quick to point out that the tool was designed to address a very specific subset of SQL Injection vulnerability — the type affected by the mass attacks — and is not designed to be a general purpose replacement for existing SQL Injection scanners. Let’s look at the limitations, as outlined on the HP page, one by one.

Limitation: Will only crawl up to 1500 pages

Depends on what they mean by 1500 pages. For example, if I have these links on my front page, is that one URL or three?

• http://www.veracode.com/blog/?p=111&foo=1
• http://www.veracode.com/blog/?p=111&foo=2
• http://www.veracode.com/blog/?p=111&foo=3

Or, does it mean that it will really only crawl 1500 pages total, so if I have the same link 1500 times on the front page, it won’t go any further? Either way, for most smaller websites this is probably fine. If you need more than 1500 you could give it different starting URLs in an attempt to improve coverage. It would be nice to have a clearer definition of what it means to “crawl up to 1500 pages” though.

Limitation: Does not support sites requiring authentication

Well, this will render it useless for the majority of enterprise apps. But there are still a lot of sites out there that don’t require authentication, including some of the ones that got hit during the mass attacks, such as the United Nations, UK government, etc.

[Update 06/26: Thomas Ptacek Mike Tracy investigates further and provides a workaround that'll work for the majority of sites that use cookie-based auth]

Limitation: Does not perform Blind SQL injection

They have taken a lot of flack for this but Billy describes it as a conscious choice:

An early version of the tool checked for blind SQL injection, but the final verison of Scrawlr did not. … The biggest feedback we got from early testing was developers wanted to “see” the vulnerability. Differential analysis is kind of difficult to visualize in a way that is helpful for the average dev, and pulling the table names through blind was too much of a performance issue.

I can sort of understand this rationale. Blind SQL Injection testing is much more susceptible to false positives. As users of any commercial web scanner or source code analyzer will attest, the more time you spend chasing down FPs, the less likely you are to put any faith in future results. It’d be nice if there was a way to toggle Blind SQL Injection testing on and off, though (could be off by default so nobody gets confused).

Limitation: Cannot retrieve database contents

Who cares? Find and fix the vulnerability. Pulling down the entire database “because you can” is a total ego move.

Limitation: Does not support JavaScript or flash parsing

Nobody does this very well anyway, particularly the JavaScript part. Writing a great crawler is probably the hardest part of writing an automated web scanner and it’s one of the biggest differentiators from one product to the next. You’re not going to get that for free.

Limitation: Will not test forms for SQL Injection (POST Parameters)

This is probably the toughest one to swallow. It’s not that difficult to parse out forms from HTML, and form POSTs can represent a major chunk of the attack surface. Granted, the Chinese tool associated with the mass attacks did operate solely on GET requests (i.e. parameters in the query string) so HP can defend this again by saying the tool is really aimed at the sites being targeted by the mass attacks. I think it’s a little short-sighted though; chances are that the mass attacks will evolve and it’s better to be proactive about it than reactive.

Conclusion

It’s tough to bash someone for releasing a free tool. I personally think HP should add an option for enabling Blind SQL Injection testing, and that they should consider supporting POSTs as well as GETs. You’re basically getting a (massively) stripped-down WebInspect for free, so take it for what it is. No single tool is a panacea.

The jury is still out on how effective Scrawlr is against the things it does claim support for. Keep watching the Web Security list; the reviews are filtering in.

Veracode Security Solutions

Security Alternatives

Security Threat Guides