Easy to Fix vs. Easy to Eradicate

Let’s start with some terminology to make sure we’re all on the same page. Sometimes people will say XSS is “not easy to fix” but what they really mean is that it’s “not easy to eradicate.” Big difference, right? Not many vulnerability classes are easy to eradicate. Take buffer overflows as an example. Buffer overflows were first documented in the early 1970s and began to be exploited heavily in the 1990s. We understand exactly how and why they occur, yet they are far from extinct. Working to eradicate an entire vulnerability class is a noble endeavor, but it’s not remotely pragmatic for businesses to wait around for it to happen. We can bite off chunks through OS, API, and framework protections, but XSS or any other vulnerability class isn’t going to disappear completely any time soon. So in the meantime, let’s focus on the “easy to fix” angle because that’s the problem developers and businesses are struggling with today.

It’s my belief that most XSS vulnerabilities can be fixed easily. Granted, it’s not as trivial as wrapping a single encoding mechanism around any user-supplied input used to construct web content, but once you learn how to apply contextual encoding, it’s really not that bad, provided you grok the functionality of your own web application. An alarming chunk of reflected XSS vulnerabilities are trivial, reading the value of a GET/POST parameter and writing it directly to an HTML page. Plenty of others are only marginally more complicated, such as retrieving a user-influenced value from the database and writing it into an HTML attribute. I contend both of these examples are easy for a developer to fix; tell me if you disagree. Basic XSS vulnerabilities like these are still very prevalent.

Of course, there are edge cases. Take this freakish example, which combines browser-specific parsing behavior with the ill-advised use of tainted input in Javascript code. Exceptions will always exist, but that doesn’t change the fact that most XSS flaws are straightforward to fix. We can take a huge bite out of the problem by eliminating these basic reflected cases, just like we started attacking buffer overflows by discouraging the use of unbounded string manipulation functions. Some will claim “developers shouldn’t be responsible for writing secure code,” which is noble and idealistic but also completely impractical in this day and age. Maybe it’ll happen eventually, but in the meantime there are fires to put out. So let’s step down from those ivory towers and impose some accountability.

Ease of Fix vs. Willingness to Fix

I’ve heard the assertion that XSS vulnerabilities aren’t getting fixed because they are difficult to fix. Asking “what percentage of XSS vulnerabilities actually get fixed and deployed to production?” is a valuable metric for the business, but it doesn’t reflect the actual difficulty of fixing an XSS vulnerability. It conflates the technical complexity with other excusesreasons why website vulnerabilities are not fixed.

At Veracode, we collected data in our State of Software Security Vol. 2 report that reveals developers are capable of fixing security issues quickly. While our data isn’t granular enough to state exactly how long it took to fix a particular flaw, we do know that in cases where developers did choose to remediate flaws and rescan, they reached an “acceptable” level of security in an average of 16 days. This isn’t to say that every XSS was eliminated, but it suggests that most were (more details on our scoring methodology can be found in the appendix of the report).

WhiteHat’s Fall 2010 study shows that nearly half of XSS vulnerabilities are fixed, and that doing so takes their customers an average of 67 days. These numbers differ from ours — particularly with regard to the number of days — but I think that can be attributed to prioritization. Perhaps fixing the XSS vulnerability didn’t rise to the top of the queue until day 66. Again, that’s more an indication that the business isn’t taking XSS seriously than it is of the technical sophistication required to fix.

At Veracode, we see thousands — sometimes tens of thousands — of XSS vulnerabilities a week. Many are of the previously described trivial variety that can be fixed with a single line of code. Some of our customers upload a new build the following day; others never do. Motivation is clearly a factor. Think about the XSS vulnerabilities that hit highly visible websites such as Facebook, Twitter, MySpace, and others. Sometimes those companies push XSS fixes to production in a matter of hours! Are their developers really that much better? Of course not. The difference is how seriously the business takes it. When they believe it’s important, you can bet it gets fixed.

Manufactured Contempt

There’s a growing faction that believes security practitioners are not qualified to comment on the difficulty of security fixes (XSS or otherwise) because we’re not the ones writing the code. The ironic thing is that this position is most loudly voiced by people in the infosec community! It’s like they are trying to be the “white knights”, coddling the poor, fragile developers so their feelings aren’t hurt. Who are we to speak for them? I find the entire mindset misguided at best, disingenous and contemptuous at worst. To be fair, Dinis isn’t the only one who has expressed this view, he’s just the straw that broke the camel’s back, so to speak. You know who you are.

Look, the vast majority of security professionals aren’t developers and never have been (notable exceptions include Christien Rioux, HD Moore, Halvar Flake, etc.). Trust me, we know it. I’ve written lots of code that I’d be horrified for any real developer to see. My stuff may be secure, but I’d hate to be the guy who has to maintain, extend, or even understand it. Here’s the thing — even though I can guarantee you I’d be terrible as a developer, most XSS flaws are so simple that even a security practioner like me could fix them! Here’s another way of looking at it: developers solve problems on a daily basis that are much more complex than patching an XSS vulnerability. Implying that fixing XSS is “too hard” for them is insulting!

That being said, who says we’re not qualified to comment on a code-level vulnerability if we’re not the one writing the fix? In fact, who’s to say that the security professional isn’t more qualified to assess the difficulty in some situations? Specifically, if a developer doesn’t understand the root cause, how can he possibly estimate the effort to fix? I’ve been on readouts where developers claim initially that several hundred XSS flaws will take a day each to fix, but then once they understand how simple it is they realize they can knock them all out in a week. Communication and education go a long way. Sure, sometimes there are complicating factors involved that affect remediation time, but I can’t recall a time where a developer has told me my estimate was downright unreasonable.

Bottom line: By and large, I don’t think developers feel miffed or resentful when we try to estimate the effort to fix a vulnerability. They know that what we say isn’t the final word, it’s simply one input into a more complex equation. Yes, developers do get annoyed when it seems like the security group is creating extra work for them, but that’s a different discussion altogether.

Ceteris Paribus

One final pet peeve of mine is the rationalization that security vulnerabilities take longer to fix because you have to identify the root cause, account for side effects, test the fix, and roll it into a either a release or a patch. As opposed to other software bugs where fixes are accomplished by handwaving and magic incantations? Of course not; these steps are common to just about any software bug. In fact, I’d argue that identifying the root cause of a security vulnerability is much easier than hunting down an unpredictable crash, a race condition, or any other non-trivial bug. Come to think of it, testing the fix may be easier too, at least compared to a bug that’s intermittent or hard to reproduce. As for side effects and other QA testing, this is why we have regression suites! If you build software and you don’t have the capability to run an automated regression suite after fixing a bug, then let’s face it, you’ve got bigger problems than wringing out a few XSS vulnerabilities.

My high school economics teacher used the term “ceteris paribus” at least once per lecture. Loosely translated from Latin, it means “all other things being equal” and it’s often used in economics and philosophy to enable one to describe outcomes without having to account for other complicating factors. The ceteris paribus concept doesn’t apply perfectly to this situation, but it’s close enough for a blog post, to wit: ceteris paribus, fixing a security-related bug is no more difficult than fixing any other critical software bug. Rattling off all the steps involved in deploying a fix is just an attempt at misdirection.

Closing Thoughts

My hope in writing this post is to spur some debate around some of the reasons, excuses, and rationalizations that often accompany the surprisingly-divisive topic of XSS. I want to hear from both security practitioners and developers on where you think I’ve hit or missed the mark. We don’t censor comments here, but there is a moderation queue, so bear with us if your comment takes a few hours to show up.

Veracode Security Solutions

Veracode Security Threat Guides

According to the statement, Kusznirewicz was playing an OLG slot machine called Buccaneer at Georgian Downs in Innisfil, Ont., on Dec. 8 when it showed he had won $42.9 million.

When the machine’s winning lights and sounds were activated, an OLG floor attendant initially told Kusznirewicz to go to the “winners circle” to claim his prize, according to the statement. But other OLG employees immediately arrived and told him that the corporation would not be paying, because there had been a “machine malfunction.”

They offered him a free dinner for four at the casino’s buffet.

In a press release, OLG described the malfunction as follows:

“The single Buccaneer-themed slot machine in question is a two cent per play machine with a base game reward of $300 and an absolute maximum payout of $9,025,” the release states.

“The $42 million figure is not a possible award given this machine’s configuration and pay table settings.”

Of course the lawsuit will probably be thrown out, or OLG will settle with the guy for a lesser amount. But from a technical perspective, it’s amusing to think about what happened to cause this scenario. You can imagine the slot machine software looking something like this:

void do_spin() {
  spin_reels();
  if (winning_combination) {
    unsigned int winnings = calculate_payout_in_cents();
    send_to_display("You've won $%u!n", winnings/100);
    add_to_balance(winnings/100);
  }
}

int calculate_payout_in_cents() {
  int rv;
  if (rv = lookup_payout_amount())
    return rv;
  else
    return -1;
}

For some reason, something caused lookup_payout_amount() to return NULL, which meant calculate_payout_in_cents() returned -1, signifying an error. Then, in addition to implicitly casting the signed result to an unsigned type, do_spin() fails to check for the error condition! It assumes success and announces the payout via the slot machine’s display. In this case, the -1, represented as 0xFFFFFFFF in two’s complement, gets interpreted as an unsigned number, 4294967295, due to the implicit cast, and the display prints “You’ve won $42949672!”

Today’s lesson: remember to check your error conditions!

It sounds easy, right? Certainly it would be easy to inject the types of classic security problems that are trivially detectable. For example:

• Replace bounded string manipulation calls with unbounded ones, e.g. strncpy() becomes strcpy(), strncat() becomes strcat(), etc.
• Replace printf style calls that use %s format string specifiers, e.g. printf("%s", buf) becomes printf(buf)
• Replace scanf() calls with everybody’s favorite function, gets() (Hi AIX developers!)
• Create type mismatches and potential integer coercion issues by replacing unsigned variables with their signed counterparts

Or, on the web application side:

• Create SQL injection by replacing prepared statements with concatenated queries — not trivial but there are a limited number of database APIs, and SQL statements follow a defined syntax, so it wouldn’t be that hard
• Inject XSS by removing all calls to known output encoding routines
• Disable input validation by removing all calls to mechanisms such as regex replacement

Of course, when you start messing with input validation, you run the risk of altering the intended operation of the program. Maybe that regex replacement you removed was security-related, but on the other hand, maybe it’s performing a transformation that’s relevant to the application logic. If you didn’t care about the program actually being able to function, you could:

• Arbitrary shorten character arrays
• Use the incorrect functions to manipulate standard and/or wide strings
• Add or remove calls to free() or delete
• Swap calls to delete and delete[]
• Remove checks for null — string contents, malloc() return values, whatever you feel like
• Create off-by-one errors in loop counters and array indexes

I could go on forever but you probably get the point. The trick would be finding the ones that didn’t make the program segfault after 30 seconds of operation. Then again, is it really important that the vulnerable version of the program behaves identically to the original under normal operating circumstances? That constraint makes it more challenging; otherwise, it’s kind of a boring exercise. You could argue that it’s important if you plan to use the modified version to test a fuzzer (or other dynamic analysis tool) but not for static analysis.

Either way, you’d eventually hit the same boundaries as a vulnerability detection tool. There would still be entire classes of vulnerabilities that could not be addressed effectively. How do you create authorization bypass issues without an understanding of what’s being protected and from whom? How do you inject CSRF without knowing which functions are meaningful and which tokens/identifiers are safe to remove? And you can forget about business logic flaws entirely. Basically, it’s hard to break something so specific without a decent understanding of how that something was designed to function in the first place.

Now that my brain is sufficiently uncluttered, I can get back to doing real work.

Also, as a proof-of-concept, we had demonstrated using IDA Pro’s scripting framework to detect one of the backdoor examples that we discussed — suspicious cryptographic API calls. Specifically, it flags calls to known encryption, decryption, and/or key management functions where a constant value is passed to a specific argument position. This can help identify situations such as an application encrypting data with a hard-coded key. We had numerous requests to post the code, so here it is:

Veracode’s binary analysis technology uses similar (but more sophisticated) techniques. We build our own intermediate representation of the binary’s data flows, control flows, and range propagation which is not based on IDA Pro. We then scan that representation for backdoors in ways similar to the cryptoconst script. However, at BlackHat you’re not allowed to promote your own products/services, so it wasn’t appropriate for us to use it for demonstration purposes.

The Software Trustworthiness Framework (STF©)
by Elfriede Dustin

Recently I presented the topic “Automated Software Testing” at a Department of Homeland Security workshop on Software Assurance. Most practitioners at the workshop equated Software Assurance with Software Security Testing and one might wonder what automated software testing has to do with software assurance?

In order to achieve software trustworthiness in the limited amount of time that’s generally allowed to produce software, a combination effort of automated software testing along with security testing is required: The Software Trustworthiness Framework (STF©) is needed.

Daily we are bombarded by news media alerts of new security breaches whether at the private, state, or government level, the latest example, UCLA having to alert 800,000 to data breach. [1]

Josh Bloch, chief Java architect at Google, said in a recent statement “Regardless of how talented and meticulous a developer is, bugs and security vulnerabilities will be found in any body of code – open source or commercial. Given this inevitably, it’s critical that all developers take the time and measures to find and fix these errors.”

Developers however are strapped cranking out new features while trying to meet the often unreasonable deadlines. First-to-market is key, beating the competition is the goal. Given this dilemma, where software developers alone cannot be responsible for software assurance, we need to look to other resources to help us win the software trustworthiness battle. Who is better suited to help a developer conduct security testing than the software testing groups already in place?

In the traditional software development lifecycle, software trustworthiness is often an afterthought, and security verification and testing efforts are delayed until after the software has been developed. Meeting deadlines is key, at all cost, including that of trustworthiness, yet vulnerabilities are an emergent property of software that appear throughout the design and implementation cycles.

Currently, much of the Security testing that is done after the software has been implemented, such as paying an external party to perform security testing and issue a report, can be considered just a Band-Aid solution. It is tempting for security testing teams to focus purely on the mechanics of testing the security of a software application and pay little attention to the surrounding tasks required of a secure software development lifecycle, such as automated software testing. This is where the STF© comes into play.

It is not possible to “test” software trustworthiness into software and it is widely known that the earlier a defect is uncovered, the cheaper it is to fix.

A full lifecycle approach to software development is the only way to achieve software trustworthiness. Therefore, combining security testing with test automation, and a before, during, and after approach to software development is required.

The most effective software trustworthiness programs start at the beginning of a project, long before any program code has been written. An effective security process is one that is used throughout the SDLC and one that employs automated testing technologies.

The Automated Testing Lifecycle Methodology (ATLM) described in the book “Automated Software Testing [2]” is a structured methodology, supports the successful implementation of automating testing, has been implemented by companies throughout the world, and is recommended by various tool vendors. The ATLM approach is consistent with rapid application development efforts including engaging the user early in the development cycle. Many universities and professional software testing organizations are adopting this methodology in their software testing courses, along with commercial companies heavily invested in automated testing.

The Secure Software Development Lifecycle (SSDL) described in Chapter 3 of the book “The Art of Software Security Testing” is a structured methodology that has emerged to support the successful implementation of secure and trustworthy software. In the SSDL security issues are evaluated and addressed early in the system’s lifecycle, during business analysis, throughout the requirements phase, and during the design and development of each software build. This early involvement allows the security team to provide a quality review of the security requirements specification, attack use cases, and software design. The team also will more completely understand business needs and requirements and the risks associated with them. Finally, the team can design and architect the most appropriate system environment using secure development methods, threat-modeling efforts, and so on to generate a more secure design. Much research has been performed in the area and resulting success of using a Secure Software Development Lifecycle (SSDL). The SSDL approach to software development is also recommended by some departments in DHS, Microsoft, and other major organizations.

Amalgamating the Automated Testing Lifecycle Methodology (ATLM) together with the Secure Software Development Lifecycle (SSDL) combines automated software testing with software security testing into the Software Trustworthiness Framework.

More on the STF©:

As outlined in Figure 1, the SSDL and ATLM represent a structured approach toward implementing trustworthy software – The Software Trustworthiness Framework. This framework mirrors the benefits of modern rapid application development efforts. Such efforts engage the stakeholders early on as well as throughout analysis, design, and development of each software build, which is done in an incremental fashion.

Figure 1 The Secure Software Development Lifecycle combined with the Automated Testing Lifecycle Methodology (ATLM) results in the Software Trustworthiness Framework (STF)

The inner layer of Figure 1 describes the ATLM and has six primary processes:

• 1. Decision to Automate Testing
• 2. Test Tool Acquisition
• 3. Automated testing Introduction Process
• 4. Test Planning, Design, and Development
• 5. Execution and Management of Tests
• 6. Test Program Review and Assessment

The outer layer of Figure 1 describes the SSDL and has six primary processes that are intertwined with the ATLM:

• A. Security guidelines, rules, and regulations – Oversight
• B. Security requirements: attack use cases
• C. Architectural and design reviews/threat modeling
• D. Secure coding guidelines
• E. Black/gray/white box testing
• F. Determining exploitability

The combined processes and tools make up the Software Trustworthiness Framework (STF©).

Implementing the Software Trustworthiness Framework (STF©) will allow for repeatable and consistent verification of new releases and software patches. It will evaluate the trustworthiness from an end-to-end system perspective, and will verify that the integration of components yield a trustable system.

Automating the testing efforts will allow for quick measurements of testing completeness and allow for testing consistency. Additionally, implementing the SSDL as described in “The Art of Software Security Testing” as part of the STF© will support efforts to make software secure and together with the ATLM it can assure that it is robust and behaves as expected in while interacting with multiple software applications, as required.

Use of the STF© will allow software testers to gain confidence that a fault in one piece of software does not introduce unknowns in a set of integrated software components. It will assure that the software trustworthiness is verifiable, given the often sparse level of software specification accessibility.

The Software Trustworthiness Framework provides a structured, multi-stage approach supporting the detailed and interrelated activities required to introduce and utilize automated tools along with security testing best practices with an iterative / spiral development approach. These activities include: test design development, development and execution of test cases, development and management of test data and the test environment, documenting, tracking and closing trouble reports found during testing.