Easy to Fix vs. Easy to Eradicate

Let’s start with some terminology to make sure we’re all on the same page. Sometimes people will say XSS is “not easy to fix” but what they really mean is that it’s “not easy to eradicate.” Big difference, right? Not many vulnerability classes are easy to eradicate. Take buffer overflows as an example. Buffer overflows were first documented in the early 1970s and began to be exploited heavily in the 1990s. We understand exactly how and why they occur, yet they are far from extinct. Working to eradicate an entire vulnerability class is a noble endeavor, but it’s not remotely pragmatic for businesses to wait around for it to happen. We can bite off chunks through OS, API, and framework protections, but XSS or any other vulnerability class isn’t going to disappear completely any time soon. So in the meantime, let’s focus on the “easy to fix” angle because that’s the problem developers and businesses are struggling with today.

It’s my belief that most XSS vulnerabilities can be fixed easily. Granted, it’s not as trivial as wrapping a single encoding mechanism around any user-supplied input used to construct web content, but once you learn how to apply contextual encoding, it’s really not that bad, provided you grok the functionality of your own web application. An alarming chunk of reflected XSS vulnerabilities are trivial, reading the value of a GET/POST parameter and writing it directly to an HTML page. Plenty of others are only marginally more complicated, such as retrieving a user-influenced value from the database and writing it into an HTML attribute. I contend both of these examples are easy for a developer to fix; tell me if you disagree. Basic XSS vulnerabilities like these are still very prevalent.

Of course, there are edge cases. Take this freakish example, which combines browser-specific parsing behavior with the ill-advised use of tainted input in Javascript code. Exceptions will always exist, but that doesn’t change the fact that most XSS flaws are straightforward to fix. We can take a huge bite out of the problem by eliminating these basic reflected cases, just like we started attacking buffer overflows by discouraging the use of unbounded string manipulation functions. Some will claim “developers shouldn’t be responsible for writing secure code,” which is noble and idealistic but also completely impractical in this day and age. Maybe it’ll happen eventually, but in the meantime there are fires to put out. So let’s step down from those ivory towers and impose some accountability.

Ease of Fix vs. Willingness to Fix

I’ve heard the assertion that XSS vulnerabilities aren’t getting fixed because they are difficult to fix. Asking “what percentage of XSS vulnerabilities actually get fixed and deployed to production?” is a valuable metric for the business, but it doesn’t reflect the actual difficulty of fixing an XSS vulnerability. It conflates the technical complexity with other excusesreasons why website vulnerabilities are not fixed.

At Veracode, we collected data in our State of Software Security Vol. 2 report that reveals developers are capable of fixing security issues quickly. While our data isn’t granular enough to state exactly how long it took to fix a particular flaw, we do know that in cases where developers did choose to remediate flaws and rescan, they reached an “acceptable” level of security in an average of 16 days. This isn’t to say that every XSS was eliminated, but it suggests that most were (more details on our scoring methodology can be found in the appendix of the report).

WhiteHat’s Fall 2010 study shows that nearly half of XSS vulnerabilities are fixed, and that doing so takes their customers an average of 67 days. These numbers differ from ours — particularly with regard to the number of days — but I think that can be attributed to prioritization. Perhaps fixing the XSS vulnerability didn’t rise to the top of the queue until day 66. Again, that’s more an indication that the business isn’t taking XSS seriously than it is of the technical sophistication required to fix.

At Veracode, we see thousands — sometimes tens of thousands — of XSS vulnerabilities a week. Many are of the previously described trivial variety that can be fixed with a single line of code. Some of our customers upload a new build the following day; others never do. Motivation is clearly a factor. Think about the XSS vulnerabilities that hit highly visible websites such as Facebook, Twitter, MySpace, and others. Sometimes those companies push XSS fixes to production in a matter of hours! Are their developers really that much better? Of course not. The difference is how seriously the business takes it. When they believe it’s important, you can bet it gets fixed.

Manufactured Contempt

There’s a growing faction that believes security practitioners are not qualified to comment on the difficulty of security fixes (XSS or otherwise) because we’re not the ones writing the code. The ironic thing is that this position is most loudly voiced by people in the infosec community! It’s like they are trying to be the “white knights”, coddling the poor, fragile developers so their feelings aren’t hurt. Who are we to speak for them? I find the entire mindset misguided at best, disingenous and contemptuous at worst. To be fair, Dinis isn’t the only one who has expressed this view, he’s just the straw that broke the camel’s back, so to speak. You know who you are.

Look, the vast majority of security professionals aren’t developers and never have been (notable exceptions include Christien Rioux, HD Moore, Halvar Flake, etc.). Trust me, we know it. I’ve written lots of code that I’d be horrified for any real developer to see. My stuff may be secure, but I’d hate to be the guy who has to maintain, extend, or even understand it. Here’s the thing — even though I can guarantee you I’d be terrible as a developer, most XSS flaws are so simple that even a security practioner like me could fix them! Here’s another way of looking at it: developers solve problems on a daily basis that are much more complex than patching an XSS vulnerability. Implying that fixing XSS is “too hard” for them is insulting!

That being said, who says we’re not qualified to comment on a code-level vulnerability if we’re not the one writing the fix? In fact, who’s to say that the security professional isn’t more qualified to assess the difficulty in some situations? Specifically, if a developer doesn’t understand the root cause, how can he possibly estimate the effort to fix? I’ve been on readouts where developers claim initially that several hundred XSS flaws will take a day each to fix, but then once they understand how simple it is they realize they can knock them all out in a week. Communication and education go a long way. Sure, sometimes there are complicating factors involved that affect remediation time, but I can’t recall a time where a developer has told me my estimate was downright unreasonable.

Bottom line: By and large, I don’t think developers feel miffed or resentful when we try to estimate the effort to fix a vulnerability. They know that what we say isn’t the final word, it’s simply one input into a more complex equation. Yes, developers do get annoyed when it seems like the security group is creating extra work for them, but that’s a different discussion altogether.

Ceteris Paribus

One final pet peeve of mine is the rationalization that security vulnerabilities take longer to fix because you have to identify the root cause, account for side effects, test the fix, and roll it into a either a release or a patch. As opposed to other software bugs where fixes are accomplished by handwaving and magic incantations? Of course not; these steps are common to just about any software bug. In fact, I’d argue that identifying the root cause of a security vulnerability is much easier than hunting down an unpredictable crash, a race condition, or any other non-trivial bug. Come to think of it, testing the fix may be easier too, at least compared to a bug that’s intermittent or hard to reproduce. As for side effects and other QA testing, this is why we have regression suites! If you build software and you don’t have the capability to run an automated regression suite after fixing a bug, then let’s face it, you’ve got bigger problems than wringing out a few XSS vulnerabilities.

My high school economics teacher used the term “ceteris paribus” at least once per lecture. Loosely translated from Latin, it means “all other things being equal” and it’s often used in economics and philosophy to enable one to describe outcomes without having to account for other complicating factors. The ceteris paribus concept doesn’t apply perfectly to this situation, but it’s close enough for a blog post, to wit: ceteris paribus, fixing a security-related bug is no more difficult than fixing any other critical software bug. Rattling off all the steps involved in deploying a fix is just an attempt at misdirection.

Closing Thoughts

My hope in writing this post is to spur some debate around some of the reasons, excuses, and rationalizations that often accompany the surprisingly-divisive topic of XSS. I want to hear from both security practitioners and developers on where you think I’ve hit or missed the mark. We don’t censor comments here, but there is a moderation queue, so bear with us if your comment takes a few hours to show up.

Veracode Security Solutions

Veracode Security Threat Guides

Digging into the full report they say that 59% of the breaches involve hacking. Of those the breakdown is this:

• Application/Service layer -39%
• OS/Platform layer – 23%
• Exploit known vulnerability -18%
• Exploit unknown vulnerability – 5%
• Use of back door -15%

“Attacks targeting applications, software, and services were by far the most common technique, representing 39 percent of all hacking activity leading to data compromise. This follows a trend in recent years of attacks moving up the stack. Far from passé, operating system, platform, and server-level attacks accounted for a sizable portion of breaches. Eighteen percent of hacks exploited a specific known vulnerability while 5 percent exploited unknown vulnerabilities for which a patch was not available at the time of the attack. Evidence of re-entry via backdoors, which enable prolonged access to and control of compromised systems, was found in 15 percent of hacking-related breaches. The attractiveness of this to criminals desiring large quantities of information is obvious.”

The largest single type of breach is hacking and within that the largest type is application/service layer attacks. So if we multiply 59% times 39% we get 23% of those 500, or 115, data breaches are due attackers hacking applications. That is a very significant number of the whole slice of the data breach pie. It is clear that securing applications is a significant part of protecting against data breaches.

Full Report

Veracode Security Solutions

Veracode Security Threat Guides

This recommendation backs up the decision made by our CTO, Chris Wysopal, more than a year ago to adopt the CVSS standard as a part of the Veracode rating system.

Another interesting recommendation in the report is: “Enterprieses should ensure that processes are in place to detect, assess, and manage each software vulnerability class.” You’ll need a combination of static, dynamic and manual testing to do it all.

Gartner requires you to have a login to read the entire article.

On a side note, we are now linking to Technorati:
Technorati Profile

PA-DSS

The PA-DSS is basically an extension of Visa’s Payment Application Best Practices (PABP) guidelines. Essentially, it’s a standard that will be applied to any application that stores, processes, or transmits cardholder data in a merchant or service provider environment. This is not to be confused with the PCI-DSS, which applies to entities/companies as opposed to applications. The PCI Council will release a draft of the PA-DSS in October for comments and suggestions, leading to a final version by the end of Q4. They are currently working on the certification process for existing QPASPs (Visa’s certification for PA assessors) to become PA-QSAs by Q2 2008.

The panel outlined the changes to PABP as follows:

• Explanation/Clarification. Just providing additional text around the best practices, presumably to make them easier to understand.
• Enhancements. Things necessary to turn a Best Practices document into a standard. The panel did not go into exhaustive detail on the enhancements, but the examples given were requiring assessors to use forensic tools to examine the output of the PA, introducing stronger lab requirements, and requiring a penetration test of the PA to identify common (e.g. OWASP-style) vulnerabilities.

Unfortunately, this was the extent of the detail. I wanted more information on the enhancements but didn’t get a chance to ask a question due to time constraints. Specifically, it would be nice to understand whether code analysis would be an alternate option for the penetration test requirement. Or what exactly they meant by using forensic tools to examine the application’s output.

Other relevant questions that came up had to do with versioning and recertification. For example, if a PA vendor releases a minor update, what is required in order to recertify the application? Would they be required to conduct another pen test, even if all they did was add a minor feature, such as the ability to recognize a new type of gift card? The panel’s response was that if a recompile was required then recertification would be necessary. However, that recertification effort could be targeted towards the new functionality. The other good question was under what lab conditions would the application be tested — the vendor or the customer. If you consider applications such as SAP, PeopleSoft, etc. which tend to be highly customized for each customer install, does it make sense to test in a vendor lab only, with no context of how the application will be deployed and configured in a production environment?

Requirement 6.6

Requirement 6.6 of the PCI-DSS becomes mandatory in June 2008 and requires all web-facing applications to either undergo a code review OR be protected by a web application firewall. There were questions on what “web-facing” means, and the Council admitted that it needed rewording. This could be interpreted as any application using HTTP as a transport mechanism, or it could mean only those applications that were Internet-facing. It sounded as if they meant the latter, but they did not say so outright which was confusing. Web services applications would also be within scope. Many people were confused about the requirements for a WAF, specifically because there are no guidelines on what functionality the WAF must have, how it must be configured, etc. Joe Lindstrom, a panelist who runs Symantec’s compliance practice, outlined these features to look for when selecting a WAF:

• Layer of defense for known attacks (e.g. OWASP Top 10)
• LDAP and/or Active Directory integration
• Logging and monitoring capabilities
• Minimal performance impact

From a security perspective, I am not sure why performance impact was on the list. As long as the security requirements are met, it should be up to you to decide how much of a performance hit your application can handle. Another panelist, Dave Wichers from Aspect Security, suggested that a WAF should also do egress filtering to detect sensitive information such as track data leaving the network.

WAFs can be configured in so many different ways, however, that people were really looking for detailed guidelines on what would satisfy the requirement. First, WAFs can be run in “detect” or “prevent” mode. The former simply logs anomalies, while the latter actively blocks traffic once it detects those anomalies. Obviously, this is a big difference from an operational risk perspective, and most organizations use “detect” mode for that reason. Secondly, the effectiveness of a WAF is limited if it is not tuned specifically for the web application that it is protecting. In order for it to know what constitutes malicious traffic, it needs to understand what normal application traffic looks like. It’s kind of like tuning an IDS, except at the application layer. Of course, this means that a certain amount of retraining is required whenever the application is updated or changed, so again, WAFs are commonly deployed without proper tuning in order to avoid unforeseen disruptions.

One panelist, Kennet Westby from Coalfire Systems, pointed out that Requirement 6.6 should not be an either-or option, since code review is an integral part of the SDLC. This time, I made it up to the microphone before time expired, and asked if the council had considered making code review the sole requirement but allowing a WAF to be used as a compensating control. In my view, this seemed more closely aligned with the intent of PCI as a security standard. Incorporating code review into the development process is the best practice, whereas a WAF is essentially just a band-aid, not a replacement for code review. Granted, a SDLC cannot be put in place overnight, and a WAF can be effective as quick fix — something is better than nothing — but it certainly shouldn’t be an equivalent way to satisfy the requirement. Dave Wichers responded to my question, agreeing that code review was the “right way,” but added that a code review is not a viable option for many companies, thus they had to offer WAF as an alternative. Time ran out, so I’m still not clear exactly what he meant by that, and I need to follow up to get more clarity on his comment. Still, I saw some heads nodding and received some positive feedback from a couple people in the audience, so I guess I’m not the only one thinking along these lines.

The panel also brought up the question of whether automated code analysis solutions could be used to satisfy the code review requirement. The general consensus on stage was that you still needed a security professional in the loop due to the false positive (FP) and false negative (FN) problem. This isn’t to say that companies shouldn’t use these tools to help them achieve compliance, but that the tools alone would not satisfy the requirement. While the panel did not recommend any specific products, they suggested that organizations should focus on three main factors when selecting code analysis vendors:

• Maturity of the product
• Ease of integration into the SDLC
• FP and FN rates as an accuracy benchmark

The idea of benchmarking for FP/FN rates shows a level of maturity that this industry is undergoing. It would be an eye-opening exercise to apply these standards to human code reviewers and human penetration testers as well. In fact, I think people would be shocked when they realize the FN levels they are accepting today.

Staying on the topic of benchmarking for a moment, the WAF discussion is interesting. What percentage of attacks do people think these are actually stopping? Is it 20% or 80%? I haven’t ever seen any solid data here, though anecdotally, the reviews certainly have not been glowing. This is a big of an unknown as FP/FN rates of code analysis tools. If automated code analysis is going to be held up to benchmark scrutiny, then WAFs and manual assessments need to be as well.

Conclusions

Overall, I found that the sessions were probably a little bit on the short side. At each session I attended, there was never enough time at the end to address audience questions. Many of the audience members were from companies working toward PCI compliance, so the questions tended to revolve around interpretation of the PCI language and intent (Is such-and-such considered a compensating control? What do you mean by connected entities?), leaving less time for discussion of the technical merits of the security requirements.

Veracode Security Solutions

Veracode Security Threat Guides

The scans include a combination of raw scan results and results that have been manually validated to remove false positive results. The statistics do not include the results of any purely manual security audits (aka human assessments).

As with any statistical data, the results of this study should be digested with a healthy dose of skepticism and a solid understanding of the sampling bias. Take, for example, a political tracking poll conducted by phone during normal business hours. The results of the poll will only account for the opinions of voters with publicly listed phone numbers who happen to be home during the day (and who don’t screen their calls to weed out tracking polls). The sampling bias of the WASC study is that it only accounts for the findings of automated web application scanners. As a result, it primarily reflects the capabilities and limitations of these scanners, not the general state of web application security, as one might reasonably expect from a WASC publication.

Keeping this bias in mind, what does this data really tell us, beyond the fact that automated vulnerability scanners find a lot of XSS? Does it give us true visibility into the actual prevalence and distribution of vulnerabilities in custom web applications? My answer is no.

Let’s look at a sample of the prevalence data:

• 100,059 instances of XSS across 26,531 web apps
• 23 instances of Insufficient Authorization across 4 web apps
• 84 instances of Insufficient Authentication, all occurring in 1 web app
• 426 instances of Path Traversal across 374 web apps

Those numbers just don’t pass the “giggle test.” The category that stands out the most in that list is Insufficient Authorization, a very common vulnerability in my experience. It’s highly unlikely that only four of the applications contain authorization-related vulnerabilities. All this does is highlight the limitations of automated web app scanners.

What about Cross-Site Request Forgery? That doesn’t show up at all on the list, despite the fact that the vast majority of web applications are vulnerable to it (even Jeremiah agrees on this point). It’s not on the list because it isn’t something the automated scanners can detect with any degree of accuracy. For the same reason, several categories on the OWASP Top Ten aren’t even represented, such as Buffer Overflows and Denial of Service.

Now let’s talk about false positives. The methodology clearly states that the data is a mixture of raw scan output and manually validated results. Since the results are presented in aggregate, it is impossible to derive real meaning from the figures without insight into the following information:

• Which results came from which product
• Which results have been manually validated
• The historical false positive rates, by category, for each product

There is also lack of clarity around the definition of “one vulnerability.” Consider this code snippet:

Map params = request.getParameterMap();
PrintWriter pw = response.getWriter();
for (String key : params.keySet())
  for (String value : params.get(key))
    pw.println(key + "=" + value + " ");

An automated scanner might report that as 100 different XSS vulnerabilities, one for each parameter that it fuzzed. However, there is only one actual flaw in the code. This is a simplistic example, but I suspect the inflated XSS numbers are partly due to this type of accounting.

In conclusion, here are the key takeaways from this list, after accounting for all of the weaknesses inherent to the methodology and the data itself:

• Automated web app scanners find a lot of XSS and SQL Injection
• Automated web app scanners are ineffective at finding vulnerabilities that require some understanding of higher-level logic, e.g. Insufficient Authorization or CSRF
• Including raw scan results from a category of products that are notorious for high false positive rates makes the resulting statistics even less meaningful
• The many-to-one mapping of vulnerabilities to actual instances of flawed code artificially inflate the prevalence of certain categories

In other words, this study provides minimal value to a veteran pen tester, and is misleading to just about anyone else.

Does this ABC News story on criminals looting 401K and online trading accounts of tens of millions of dollars surprise anyone in the security field? Well of course it shouldn’t. We have known about the potential for this type of criminal activity for over 8 years.

We are performing computing that requires high assurance, such as managing an online trading account, on a low assurance terminal, the home computer. The network layer of these transactions is using high security crypto in the form of SSL. The financial institution very likely has excellent security on the servers that run their account management software. But what about the customer? Who is making sure his or her machine is up to the task of being part of a high assurance transaction? The answer is nobody. Attackers will always go for the weak link and that weak link is, by far, the end user’s computer.

I gave a presentation at the Digital Commerce Society of Boston on April 6th, 1999. It was titled, “Client Security: You’ve got armored trucks, but what about the pick pockets?” Here is the presentation abstract from the announcement:

Everyone in ecommerce these days is peddling better vaults for stores and stronger armored cars to deliver payments and merchandise. Does this really matter in an Internet world where you can pick the pocket of a consumer? Or more likely, to automate the pocket picking of a large number of consumers.

Current authentication and purchasing systems rely on consumers using off the shelf operating systems such as Windows 95/98. This is the operating system which Microsoft has admitted to having no security model. Current ecommerce client security is layering strong encryption on this bed of jello.

What are some of the attacks that are being used? What technology can be used to overcome this problem?

In June 1999 I was interviewed on Dateline NBC by Lea Thompson. The story was about how banks were moving into the world of online banking without considering the risks of the customer’s home computer. It’s as if the banks were only setting up their branches in high crime neighborhoods, knowing there were dozens of skilled pick pockets waiting outside the door, and doing nothing to protect customers.

Catherine Allen, the CEO of BITS, a consortium of 100 of the largest financial institutions, was on the program defending financial organizations and stating that online banking was secure.

The final participants in the show were 2 young men in a dark room in San Francisco that agreed to demonstrate how a Trojan keystroke logger worked. The correspondent, Lea Thompson, sat at her home computer. The only information the young men knew was her email address. They sent her an email with a Trojan animated greeting card attached. It contained a keystroke logger. She opened the attachment. She then proceeded to log into her online checking account. Then she logged out. The young men in San Francisco retrieved her password from the keystroke log and proceeded to log in and check her balance. They decided not to transfer any money because it might be considered wire fraud.

It was clear to any viewer of this Dateline NBC show 8 years ago that this was a problem. So where have we gotten in the intervening 8 years? Online crime has grown dramatically and most of the attacks are taking place on the customer’s home computer –- a threat that was clearly demonstrated to banks and customers on national television.

Symantec’s latest Internet Security Threat Report has the sobering statistics showing that the “pick pocket” risk to online banking is getting worse.

• Home users were the most highly targeted sector, accounting for 93 percent of all targeted attacks.
• Eighty-six percent of the credit and debit cards advertised for sale on underground economy servers known to Symantec were issued by banks in the United States.
• The volume of Trojans in the top 50 malicious code samples reported to Symantec increased from 23 percent to 45 percent.
• Trojans accounted for 60 percent of the top 50 malicious code samples when measured by potential infections.
• Keystroke logging threats made up 79 percent of confidential information threats by volume of reports, up from 57 percent in the first half of the year and 66 percent in the second half of 2005.
• Seventy-eight percent of malicious code that propagated did so over SMTP, making it the most commonly used propagation mechanism.

So it is clear the combination of user behavior, home computer security, email, and online banking lead to massive amounts of online theft.

Back in 1999 at my talk at the Digital Commerce Society of Boston I recommended that banks require authentication tokens for accounts with more than $2000. A token doesn’t completely solve all of these problems but it can at least cut down on one of the biggest which is keystroke logging of passwords. This recommendation was dismissed as too expensive at the time. I recommended building software more securely which is starting to become an accepted practice but it needs to become a requirement.

What did we mainly do over the last 8 years? We upgraded operating systems and browsers. We added anti-virus, personal firewalls, and anti-phishing protection. We learned not to open certain attachments. Yet the problem got worse.

It is going to take a dramatic change in what we think of as secure authentication and a dramatic change in how we build software to turn this around. If we don’t do these things I fear in 2015 we will be looking at cybercrime rates that have either gone through the roof or people will have stopped trusting the Internet to manage financial accounts.