With regard to the recent Patch Tuesday fix, there has been an issue fixed regarding NTLM Relaying, that has been around for more than eight years.

In 2000, I wrote an advisory about NTLM relaying (CVE-2000-0834). The problem turned out to be significantly larger than I originally suggested in the advisory. The attack extended to other NTLM-based authentications on other protocols and allowed general-purpose credential theft via a man-in-the-middle attack.

The SMBRelay tool was published in 2001 by Sir Dystic of Cult Of The Dead Cow, and that really took it to the next level. The protocol completely fell apart. It kicked off a number of other analyses of the NTLM protocol that finally resulted in this patch. Eight years after it’s discovery.

At least they got around to it. Thanks!

I’m finally getting around to finishing my post on minimizing attack surfaces. Here’s Part 1, in case you missed it.

First, a quick clarification. I noticed that some of the readers who commented on that first post wanted to talk about improving security through the use of various development methodologies or coding frameworks. Those are interesting tangents (and ones that I may write about in the future), but my intention with this post is to discuss a very specific problem related to how people integrate third-party code — that is, the stuff you import or link in but didn’t write yourself.

As I mentioned previously, developers have a tendency to “bolt on” third-party components to applications without understanding the security implications. Often, these components are glossed over or ignored completely during threat modeling discussions. I attempted to illustrate this with my fictitious WhizBang library example in Part 1.

When integrating a third-party component, developers familiarize themselves with the API but generally don’t care how it’s implemented. Granted, that’s how an API is supposed to work; you don’t have to futz around with code beyond the API boundary, and you can blissfully ignore parts of the library that you don’t need. In past consulting gigs, I’ve sat in threat modeling discussions where nobody knew whether a particular library generated network traffic. “We just use the API,” they say. The fact that it works is good enough; nobody seems to care how it works.

That mindset is ideal for rapid development but problematic for security. Failing to understand the complete application, as opposed to just the part you wrote, prevents you from accurately assessing its security posture.

It’s also no coincidence that web app pen testers love third-party components — we get excited when we see “bolted on” interfaces, because we know that developers tend to leave extraneous functionality exposed. The resulting findings usually generate reactions such as “I didn’t even know that servlet had an upload function.”

An Example

Here’s a close-to-home example related to my post about DWR 2.0.5 from the other day. DWR is an Ajax framework that has a variety of operating modes. In-house, we use a subset of DWR’s full functionality — specifically, we interact with it using the “plaincall” method only, so we made sure that the features we didn’t need were disabled via the configuration file. As it turned out, there were vulnerable code paths prior to the “do you have this thing disabled” check. In hindsight, if we had taken more time to understand the exposed interfaces, we could have reduced the attack surface by filtering out unneeded request patterns before they even touched the third-party code.

But wait, you say. What about maintainability? If I whitelist using a point-in-time application profile, doesn’t this create the same maintenance headache as the reviled WAF? It doesn’t have to. Certainly, one option would be to whitelist each and every unique URL that references the DWR framework, e.g.

/dwr/call/plaincall/myMethod1
/dwr/call/plaincall/myMethod2
/dwr/call/plaincall/myMethod3

But then you’d have to update the whitelist every time you added or removed functionality from your application. Also, don’t lose sight of the security goal, which is to minimize the amount of exposed third-party code. If I add or remove URLs that list, provided they are still using the “plaincall” method, I’m hitting the same DWR dispatcher every time. So I’ve increased maintenance cost without any security benefit.

A better option is to simply tighten the URL pattern a bit in the J2EE container. Here’s the default configuration:

<servlet-mapping>
  <servlet-name>dwr-invoker</servlet-name>
  <url-pattern>/dwr/*</url-pattern>
</servlet-mapping>

Now, instead of allowing every URL starting with /dwr/ to be processed by the DWR library, you could be a little more restrictive:

<servlet-mapping>
  <servlet-name>dwr-invoker</servlet-name>
  <url-pattern>/dwr/call/plaincall/*</url-pattern>
</servlet-mapping>

In this configuration, you don’t have to worry about /dwr/call/someothercodepath any more. There is less third-party code exposed, thereby reducing the overall attack surface of the application. (NB: DWR also serves up a couple of Javascript files, so those URL patterns will have to be whitelisted too)

A Logical Extension

Even if you’re not a developer, you should still be thinking about attack surfaces. People download and install blogging platforms such as WordPress, Movable Type, etc. all the time, but how many take additional steps to harden their installations? The concept is the same as the OS hardening analogy I brought up at the very beginning of this discussion.

Similarly, people install third-party WordPress plugins or Joomla components without considering that most of them are written by some random programmer who is a whiz with the plugin API but knows nothing about security?

At the risk of sounding trite, always remember that security is only as strong as the weakest link.

What was the first thing you learned about network security? There’s a good chance it had something to do with port scanning. After scanning a few boxes, you realized that modern operating systems have a lot of open ports by default, meaning a lot of services. Some had an obvious purpose, like telnet on tcp/23 or ftp fon tcp/21. Others left you wondering, what the heck is listening on tcp/515 or tcp/7100? And remember, you couldn’t ask Google because it didn’t exist (well, maybe it did depending on when you got into security).

Your first real lesson about locking down a host was how to reduce its attack surface. You learned how to disable services using /etc/inetd.conf. Then you learned about rc.d and how to prevent unnecessary services from being launched at startup. Next, maybe you configured the Xserver to disallow remote connections or moved on to removing setuid permissions from files. As you worked, you’d periodically re-scan the box to gauge progress, asking yourself “have I removed everything I don’t need?” The underlying motivation, of course, is that an attacker can’t hack something that isn’t there.

You learned how to extend those concepts to the network — configuring firewall rules, router ACLs, VLANs, etc. Segmenting the network. Creating a DMZ. No need to dwell on this, you get the idea.

Eventually, people realized that applications had an attack surface too. Web servers and application servers got a lot of attention, followed closely by custom web applications. “What do you mean you can execute SQL queries against my database? That’s impossible, I have a firewall!”

Some companies, the ones who could afford it anyway, started to build security into their development cycle. Doing threat modeling during the design phase made sense, because hey, it’s much cheaper to fix security holes in a whiteboard drawing than it is to rewrite your authorization module from scratch after it’s in production.

Let’s talk strictly about custom web applications now. What I’ve observed is that most development groups, even the ones who actively engage in threat modeling, do not understand their web application’s attack surface. The lead architect can whiteboard a high-level diagram of all the major components and how they interact. Individual developers can go a bit deeper, telling you which files they touch, what database permissions they need, or how various pieces of data are encrypted in storage. At the end of this exercise you have a complete picture of the processes, data flows, protocols, privilege boundaries, external entities, and so on, and you’re well on your way to understanding all of the potential attack vectors.

Or are you?

What often gets overlooked or glossed over is the impact of external libraries or packages. Nobody writes everything from scratch. A typical list of third-party libraries for a Java-based Web 2.0 application might include DWR, GWT, Axis, and Dojo, plus about 30 other libraries to do everything from logging to parsing to image manipulation. Nine out of ten times, the libraries will be installed in full, using the default configuration from page one of the README file.

Why is this relevant? Because just as those old Unix boxes exposed unnecessary services, libraries expose unnecessary code. Let’s say you installed Dojo to simplify the process of creating an HTML table with rows and columns that can be sorted on demand. Did you remember to remove all the .js files you didn’t need? Or maybe you installed Axis or DWR or anything else that has its own Servlet(s) for processing requests. Have you compared what that Servlet can do against what you need it to do?

A fictitious example may help illustrate further. Imagine you just downloaded a new library called WhizBang. You follow the installation instructions to define and map two servlets in your web.xml file, WhizServlet and BangServlet, and you configure it to integrate with your web app. After a bit of trial and error, it’s functional. Yay! This is where most developers stop.

Nobody asks, “how much of this do I actually need?” Case in point, what if your application only uses WhizServlet? BangServlet is still exposed, and you don’t even use it! Similarly, what if WhizServlet takes an “action” parameter which can be either “view”, “edit”, or “delete”, and your application only uses “view”? You’re still exposing the other actions to anybody who knows the URL syntax (pretty trivial if it’s open source). You wouldn’t expose large chunks of your own code that you weren’t using, so why should it be any different with libraries?

This post is getting kind of long so I’m going to split it up. In the next post, I’ll continue the discussion of attack surface minimization, as well as some of the tradeoffs that go along with this approach.

I was just reading Dre’s post, R.I.P. CISSP, over at the tssci security blog, in which he predicts the upcoming OWASP People Certification Project will be the next big thing. This paragraph is quoted from James McGovern’s blog (James is the project leader):

As an Enterprise Architect, I understand the importance of the ability for a security professional to articulate risk to IT and business executives, yet I am also equally passionate that security professionals should also have the capability to sit down at a keyboard and actually do something as opposed to just talking about [it].

I agree wholeheartedly with this sentiment, and I believe the project goals are noble. So I went to read the latest OPCP draft proposal to see how they planned to tackle this admittedly difficult problem. What did I find? It’s just another test, with questions in a dozen or so broad categories. Far more specialized that CISSP, with topics that are more relevant to application security, but ultimately, still just a test.

The comment I once made about security educators/trainers is relevant here. Whatever questions end up on the OPCP test, these educators could probably answer most of them correctly without even studying. They lecture day in and day out about these topics. They have heard obscure questions and are prepared to answer them. And yet, many of them do not have any practical field experience.

A client chastised me once for making a statement that penetration testing is a mixture of art and science. He wanted to believe that it was completely scientific and could be distilled down to a checklist type approach. I explained that while much of it can be done methodically, there is a certain amount of skill and intuition that only comes from practical experience. You learn to recognize that “gut feel” when something is amiss. He became rather incensed and, in effect, told me I was full of it. This customer went on to institute a rigid, mechanical internal process for web app pen testing that was highly inefficient and, ultimately, still relied mostly on a couple bright people on the team who were in tune with both the art and the science.

Certifications only test the science.

One of my favorite pieces of swag from RSA was this “Not a CISSP” button that was pinned onto me by none other than Sinan Eren as I was chatting with Justine Aitel at the Immunity booth. Actually, there should have been a prize awarded just for finding the Immunity booth — they were subletting another vendor’s space for a few hours at a time, so one minute they’d be there and the next they were gone.

I digress. What inevitably happened once I started walking around with this button proudly displayed was that I would get one of two reactions. The first group — mostly current and former co-workers and acquaintances — understood the humor and got a good chuckle out of it. The second group would ponder for a bit and then ask, with some confusion, why I’d intentionally point out the fact that I’m not a CISSP. I’d give a brief answer and get back to talking about Veracode (we booth babes have responsibilities, you know).

So, why indeed? The long answer is that like many security certifications, it’s an ineffective measure of a security professional’s practical abilities. Employers and customers often assume the guy with the five magic letters on his resume is technically superior to the guy without. In my experience, it’s exactly the opposite, particularly in situations where you have to sit down at a keyboard and actually DO something as opposed to talking about it. Certainly, I’ve encountered some very notable exceptions to this observation, but we’re playing by the 80/20 rule here.

There’s a good reason for this. The trend in information security is toward specialization. Security has become such a broad umbrella of varying disciplines that it’s quite difficult to be a generalist. A security career is a balance between breadth and depth, and these days, the skilled pen tester, reverse engineer, or vulnerability researcher is more marketable than the guy who knows a little bit about dozens of different disciplines but can’t apply that knowledge in a practical situation. The CISSP subject matter illustrates this perfectly — you have cryptographic algorithms, site location principles, network security, and civil law on the same exam. I won’t even get into the complaints I’ve heard about the poorly-worded, overly simplistic exam questions or the ones that simply test one’s ability to memorize obscure facts.

I’m not claiming that there’s no value to holding the CISSP certification. It can’t hurt to have some exposure to business continuity planning, for example. The problem, as I stated in the beginning, is that the CISSP title is often interpreted as an indicator of practical abilities rather than a book-level understanding of security basics. These misaligned expectations can ultimately lead to bad hiring or staffing decisions.

Career advice, take it or leave it: If an employer or prospective employer demands that you get your CISSP in order to be hired or to progress in your career, run fast in the opposite direction and find a place where you will be valued for your cumulative experience rather than a piece of paper. Learn by doing, don’t “learn the test,” so to speak.

And that, in a nutshell, is why I love my “Not a CISSP” button.

By the way, here was my other favorite from RSA, thanks to WhiteHat. This one and “Samy is my hero” were the best out of a pretty clever selection… even though they forgot the semicolon after the single quote. <grin>

So it seems that SquirrelMail 1.4.11 and 1.4.12 were recently backdoored. Similar to some high-profile backdoors in the past, this was done by modifying the distribution tarball on rather than infiltrating the source code repository [1]. In this case, the backdoor was detected when a user noticed that the MD5 published on SquirrelMail’s website didn’t match the calculated MD5 from the SourceForge distribution.

Since the SVN repository remained intact, we can’t go back and examine the backdoor in detail. However, we do have a newsgroup posting that sheds a little light on the situation:

> What diff do you see between the compromised version and
> the one that is there now? I see only a comment diff in one file.

it was a small block of code that checks for a $_SERVER var. If that var was present, it would redefine SM_PATH. Under normal circumstances, this would never be executed, but we have since learned how to make it execute.

In PHP, $_SERVER is an array populated by the web server that contains information such as headers, paths, and script locations. This includes some user-supplied input such as the URL query string and the HTTP headers. SM_PATH is the filesystem path where SquirrelMail is configured to be run from. So once an attacker controls SM_PATH, it’s likely that a subsequent call to include() can be exploited to fetch and execute PHP code from a remote server. This is a typical example of a Remote File Include vulnerability.

Note that the attacker backdoored the 1.5.1 distribution as well, with the same type of vulnerability but at a different location in the codebase.

I think what’s most interesting to me about this is that so many open source projects still rely on MD5 hashes for integrity checking. The minute the Xiaoyun Wang paper on MD5 collisions was released, every security practitioner in the world considered MD5 unsafe from that point forward. Even though practical attacks had not yet been formulated, the writing was on the wall. Unfortunately, the rest of the world either didn’t notice or didn’t care.

Cryptographers have since developed increasingly sophisticated attacks stemming from Wang’s original work. Recently, researchers in the Netherlands demonstrated two examples of chosen-prefix attacks which would make it possible for an attacker to take two tarballs (one original, one backdoored) and append a series of bytes to each that result in both files having the same MD5 hash. This proves beyond a shadow of a doubt that MD5 is not an effective method for verifying software integrity. There was hardly any doubt that this attack would surface eventually, so why is MD5 still in such widespread usage?

Cryptographic weaknesses aside, a lot of people completely miss the mark with hashes. MD5 or SHA-1 (or any hash function) are not very effective if the only way a user can verify them is on the same website where the download is hosted. If the download point is compromised, chances are the attacker can modify the hashes printed on the website too. Even when it’s done correctly, hashes only help identify when the distribution point is compromised. It does nothing to protect against source code compromise or vulnerabilities in the development tool chain.

We were more than pleased to read a new report by John Pescatore of Gartner recommending that security managers adopt the use of the Common Vulnerability Scoring System (CVSS) to support more repeatable, fast-acting vulnerability management processes.

This recommendation backs up the decision made by our CTO, Chris Wysopal, more than a year ago to adopt the CVSS standard as a part of the Veracode rating system.

Another interesting recommendation in the report is: “Enterprieses should ensure that processes are in place to detect, assess, and manage each software vulnerability class.” You’ll need a combination of static, dynamic and manual testing to do it all.

Gartner requires you to have a login to read the entire article.

On a side note, we are now linking to Technorati:
Technorati Profile

Earlier this week, I attended the first PCI Community Meeting in Toronto, a gathering organized by the PCI Security Standards Council to bring QSAs, ASVs, and other PCI stakeholders together in one room with the PCI Council. Let’s be honest here — in the security industry, discussing regulatory compliance is about as dull as it gets. On the other hand, compliance is also a major catalyst, sometimes the only catalyst, in convincing organizations to improve their security posture, so it’s important to understand. As might be expected, I focused my attention on the sessions dealing with upcoming application security requirements and standards. Of particular interest were the two panel discussions covering the Payment Application Data Security Standard (PA-DSS) and Requirement 6.6 of the PCI Data Security Standard (PCI-DSS).

PA-DSS

The PA-DSS is basically an extension of Visa’s Payment Application Best Practices (PABP) guidelines. Essentially, it’s a standard that will be applied to any application that stores, processes, or transmits cardholder data in a merchant or service provider environment. This is not to be confused with the PCI-DSS, which applies to entities/companies as opposed to applications. The PCI Council will release a draft of the PA-DSS in October for comments and suggestions, leading to a final version by the end of Q4. They are currently working on the certification process for existing QPASPs (Visa’s certification for PA assessors) to become PA-QSAs by Q2 2008.

The panel outlined the changes to PABP as follows:

• Explanation/Clarification. Just providing additional text around the best practices, presumably to make them easier to understand.
• Enhancements. Things necessary to turn a Best Practices document into a standard. The panel did not go into exhaustive detail on the enhancements, but the examples given were requiring assessors to use forensic tools to examine the output of the PA, introducing stronger lab requirements, and requiring a penetration test of the PA to identify common (e.g. OWASP-style) vulnerabilities.

Unfortunately, this was the extent of the detail. I wanted more information on the enhancements but didn’t get a chance to ask a question due to time constraints. Specifically, it would be nice to understand whether code analysis would be an alternate option for the penetration test requirement. Or what exactly they meant by using forensic tools to examine the application’s output.

Other relevant questions that came up had to do with versioning and recertification. For example, if a PA vendor releases a minor update, what is required in order to recertify the application? Would they be required to conduct another pen test, even if all they did was add a minor feature, such as the ability to recognize a new type of gift card? The panel’s response was that if a recompile was required then recertification would be necessary. However, that recertification effort could be targeted towards the new functionality. The other good question was under what lab conditions would the application be tested — the vendor or the customer. If you consider applications such as SAP, PeopleSoft, etc. which tend to be highly customized for each customer install, does it make sense to test in a vendor lab only, with no context of how the application will be deployed and configured in a production environment?

Requirement 6.6

Requirement 6.6 of the PCI-DSS becomes mandatory in June 2008 and requires all web-facing applications to either undergo a code review OR be protected by a web application firewall. There were questions on what “web-facing” means, and the Council admitted that it needed rewording. This could be interpreted as any application using HTTP as a transport mechanism, or it could mean only those applications that were Internet-facing. It sounded as if they meant the latter, but they did not say so outright which was confusing. Web services applications would also be within scope. Many people were confused about the requirements for a WAF, specifically because there are no guidelines on what functionality the WAF must have, how it must be configured, etc. Joe Lindstrom, a panelist who runs Symantec’s compliance practice, outlined these features to look for when selecting a WAF:

• Layer of defense for known attacks (e.g. OWASP Top 10)
• LDAP and/or Active Directory integration
• Logging and monitoring capabilities
• Minimal performance impact

From a security perspective, I am not sure why performance impact was on the list. As long as the security requirements are met, it should be up to you to decide how much of a performance hit your application can handle. Another panelist, Dave Wichers from Aspect Security, suggested that a WAF should also do egress filtering to detect sensitive information such as track data leaving the network.

WAFs can be configured in so many different ways, however, that people were really looking for detailed guidelines on what would satisfy the requirement. First, WAFs can be run in “detect” or “prevent” mode. The former simply logs anomalies, while the latter actively blocks traffic once it detects those anomalies. Obviously, this is a big difference from an operational risk perspective, and most organizations use “detect” mode for that reason. Secondly, the effectiveness of a WAF is limited if it is not tuned specifically for the web application that it is protecting. In order for it to know what constitutes malicious traffic, it needs to understand what normal application traffic looks like. It’s kind of like tuning an IDS, except at the application layer. Of course, this means that a certain amount of retraining is required whenever the application is updated or changed, so again, WAFs are commonly deployed without proper tuning in order to avoid unforeseen disruptions.

One panelist, Kennet Westby from Coalfire Systems, pointed out that Requirement 6.6 should not be an either-or option, since code review is an integral part of the SDLC. This time, I made it up to the microphone before time expired, and asked if the council had considered making code review the sole requirement but allowing a WAF to be used as a compensating control. In my view, this seemed more closely aligned with the intent of PCI as a security standard. Incorporating code review into the development process is the best practice, whereas a WAF is essentially just a band-aid, not a replacement for code review. Granted, a SDLC cannot be put in place overnight, and a WAF can be effective as quick fix — something is better than nothing — but it certainly shouldn’t be an equivalent way to satisfy the requirement. Dave Wichers responded to my question, agreeing that code review was the “right way,” but added that a code review is not a viable option for many companies, thus they had to offer WAF as an alternative. Time ran out, so I’m still not clear exactly what he meant by that, and I need to follow up to get more clarity on his comment. Still, I saw some heads nodding and received some positive feedback from a couple people in the audience, so I guess I’m not the only one thinking along these lines.

The panel also brought up the question of whether automated code analysis solutions could be used to satisfy the code review requirement. The general consensus on stage was that you still needed a security professional in the loop due to the false positive (FP) and false negative (FN) problem. This isn’t to say that companies shouldn’t use these tools to help them achieve compliance, but that the tools alone would not satisfy the requirement. While the panel did not recommend any specific products, they suggested that organizations should focus on three main factors when selecting code analysis vendors:

• Maturity of the product
• Ease of integration into the SDLC
• FP and FN rates as an accuracy benchmark

The idea of benchmarking for FP/FN rates shows a level of maturity that this industry is undergoing. It would be an eye-opening exercise to apply these standards to human code reviewers and human penetration testers as well. In fact, I think people would be shocked when they realize the FN levels they are accepting today.

Staying on the topic of benchmarking for a moment, the WAF discussion is interesting. What percentage of attacks do people think these are actually stopping? Is it 20% or 80%? I haven’t ever seen any solid data here, though anecdotally, the reviews certainly have not been glowing. This is a big of an unknown as FP/FN rates of code analysis tools. If automated code analysis is going to be held up to benchmark scrutiny, then WAFs and manual assessments need to be as well.

Conclusions

Overall, I found that the sessions were probably a little bit on the short side. At each session I attended, there was never enough time at the end to address audience questions. Many of the audience members were from companies working toward PCI compliance, so the questions tended to revolve around interpretation of the PCI language and intent (Is such-and-such considered a compensating control? What do you mean by connected entities?), leaving less time for discussion of the technical merits of the security requirements.

Network World recently published an article entitled Cisco says FTP feature in IOS is a hacker backdoor. The opening paragraph reads as follows:

Cisco says a flaw in the FTP server utility in its IOS router/switch software could be used as a backdoor by attackers.

Do you see the discrepancy? The opening statement is inconsistent with the title of the article. Are they saying that the flaw could be used as a backdoor, or that the flaw itself is a backdoor?

Any vulnerability that is remote, pre-authentication, and trivial to exploit could be used as a backdoor of sorts, once details emerge and somebody publishes an exploit script that the kiddies can use. This would clearly be a big story, considering the sheer number of Cisco routers that will require patching. However, if the flaw itself is a backdoor, that implies that the vulnerability was introduced maliciously, either by an insider or by someone who broke into Cisco’s network and modified the IOS codebase. This would be a HUGE story, because now you have to wonder about the extent to which the IOS codebase has been compromised.

First let’s break down the Cisco advisory. Like most vendor advisories, you don’t get many technical details. However, here is what we do know:

• A remote attacker can bypass the IOS FTP server authentication mechanism
• Having bypassed authentication, the attacker can read or write any file on the router’s filesystem, including the startup-config file
• The vulnerability gets the maximum possible CVSS rating of 10.0 because it is remote, pre-authentication, and trivially exploitable, with full confidentiality, integrity, and availability impact

But wait, what’s this? There’s also a Denial of Service vulnerability? Apparently, you can make IOS reload through the FTP interface, and repeatedly triggering this reload (think reboot) could effectively create an outage. The important part is that there’s a way to force a reload.

The advisory also says that a remote attacker could “cause the affected device to reload, or execute arbitrary code” but I’m not so sure about the arbitrary code part. Maybe they just mean arbitrary IOS commands.

Basically, an attacker could take control of the router a couple different ways. Network security guys, correct me if I am off base here or omitting any good attack vectors:

• Fast but clumsy: Retrieve the startup-config, alter the commands used to set the line password and enable password, upload the modified startup-config, force a reload, and login using the new passwords
• Slower but less obvious: Retrieve the startup-config, crack the passwords (difficulty dependent on whether they are configured as “passwords” or “secrets”), and login using the cracked passwords

Again, the vulnerability is undoubtedly a big deal, in and of itself. But what I really want to know is whether this was an intentional backdoor planted in the codebase or just a horribly dumb implementation error. The ability to not only bypass authentication but also force a reload seems a little suspicious, but we won’t know for sure unless Cisco discloses more information. More importantly, if it does turn out to be a bonafide backdoor, what does this say about Cisco’s SDLC?

[Today we have our first guest blog entry from Elfriede Dustin. Elfriede is a co-author of "The Art of Software Security Testing" and has written a few books on software testing, most notably, "Automated Software Testing" published by Addison-Wesley in 1999. We have heard plenty from security experts on how to fix the software development process to produce more secure software. Elfriede brings a QA practitioners viewpoint. I'd like to hear more from the testing community on this topic. - Chris Wysopal]

The Software Trustworthiness Framework (STF©)
by Elfriede Dustin

Recently I presented the topic “Automated Software Testing” at a Department of Homeland Security workshop on Software Assurance. Most practitioners at the workshop equated Software Assurance with Software Security Testing and one might wonder what automated software testing has to do with software assurance?

In order to achieve software trustworthiness in the limited amount of time that’s generally allowed to produce software, a combination effort of automated software testing along with security testing is required: The Software Trustworthiness Framework (STF©) is needed.

Daily we are bombarded by news media alerts of new security breaches whether at the private, state, or government level, the latest example, UCLA having to alert 800,000 to data breach. [1]

Josh Bloch, chief Java architect at Google, said in a recent statement “Regardless of how talented and meticulous a developer is, bugs and security vulnerabilities will be found in any body of code – open source or commercial. Given this inevitably, it’s critical that all developers take the time and measures to find and fix these errors.”

Developers however are strapped cranking out new features while trying to meet the often unreasonable deadlines. First-to-market is key, beating the competition is the goal. Given this dilemma, where software developers alone cannot be responsible for software assurance, we need to look to other resources to help us win the software trustworthiness battle. Who is better suited to help a developer conduct security testing than the software testing groups already in place?

In the traditional software development lifecycle, software trustworthiness is often an afterthought, and security verification and testing efforts are delayed until after the software has been developed. Meeting deadlines is key, at all cost, including that of trustworthiness, yet vulnerabilities are an emergent property of software that appear throughout the design and implementation cycles.

Currently, much of the Security testing that is done after the software has been implemented, such as paying an external party to perform security testing and issue a report, can be considered just a Band-Aid solution. It is tempting for security testing teams to focus purely on the mechanics of testing the security of a software application and pay little attention to the surrounding tasks required of a secure software development lifecycle, such as automated software testing. This is where the STF© comes into play.

It is not possible to “test” software trustworthiness into software and it is widely known that the earlier a defect is uncovered, the cheaper it is to fix.

A full lifecycle approach to software development is the only way to achieve software trustworthiness. Therefore, combining security testing with test automation, and a before, during, and after approach to software development is required.

The most effective software trustworthiness programs start at the beginning of a project, long before any program code has been written. An effective security process is one that is used throughout the SDLC and one that employs automated testing technologies.

The Automated Testing Lifecycle Methodology (ATLM) described in the book “Automated Software Testing [2]” is a structured methodology, supports the successful implementation of automating testing, has been implemented by companies throughout the world, and is recommended by various tool vendors. The ATLM approach is consistent with rapid application development efforts including engaging the user early in the development cycle. Many universities and professional software testing organizations are adopting this methodology in their software testing courses, along with commercial companies heavily invested in automated testing.

The Secure Software Development Lifecycle (SSDL) described in Chapter 3 of the book “The Art of Software Security Testing” is a structured methodology that has emerged to support the successful implementation of secure and trustworthy software. In the SSDL security issues are evaluated and addressed early in the system’s lifecycle, during business analysis, throughout the requirements phase, and during the design and development of each software build. This early involvement allows the security team to provide a quality review of the security requirements specification, attack use cases, and software design. The team also will more completely understand business needs and requirements and the risks associated with them. Finally, the team can design and architect the most appropriate system environment using secure development methods, threat-modeling efforts, and so on to generate a more secure design. Much research has been performed in the area and resulting success of using a Secure Software Development Lifecycle (SSDL). The SSDL approach to software development is also recommended by some departments in DHS, Microsoft, and other major organizations.

Amalgamating the Automated Testing Lifecycle Methodology (ATLM) together with the Secure Software Development Lifecycle (SSDL) combines automated software testing with software security testing into the Software Trustworthiness Framework.

More on the STF©:

As outlined in Figure 1, the SSDL and ATLM represent a structured approach toward implementing trustworthy software - The Software Trustworthiness Framework. This framework mirrors the benefits of modern rapid application development efforts. Such efforts engage the stakeholders early on as well as throughout analysis, design, and development of each software build, which is done in an incremental fashion.

Figure 1 The Secure Software Development Lifecycle combined with the Automated Testing Lifecycle Methodology (ATLM) results in the Software Trustworthiness Framework (STF)

The inner layer of Figure 1 describes the ATLM and has six primary processes:

• 1. Decision to Automate Testing
• 2. Test Tool Acquisition
• 3. Automated testing Introduction Process
• 4. Test Planning, Design, and Development
• 5. Execution and Management of Tests
• 6. Test Program Review and Assessment

The outer layer of Figure 1 describes the SSDL and has six primary processes that are intertwined with the ATLM:

• A. Security guidelines, rules, and regulations - Oversight
• B. Security requirements: attack use cases
• C. Architectural and design reviews/threat modeling
• D. Secure coding guidelines
• E. Black/gray/white box testing
• F. Determining exploitability

The combined processes and tools make up the Software Trustworthiness Framework (STF©).

Implementing the Software Trustworthiness Framework (STF©) will allow for repeatable and consistent verification of new releases and software patches. It will evaluate the trustworthiness from an end-to-end system perspective, and will verify that the integration of components yield a trustable system.

Automating the testing efforts will allow for quick measurements of testing completeness and allow for testing consistency. Additionally, implementing the SSDL as described in “The Art of Software Security Testing” as part of the STF© will support efforts to make software secure and together with the ATLM it can assure that it is robust and behaves as expected in while interacting with multiple software applications, as required.

Use of the STF© will allow software testers to gain confidence that a fault in one piece of software does not introduce unknowns in a set of integrated software components. It will assure that the software trustworthiness is verifiable, given the often sparse level of software specification accessibility.

The Software Trustworthiness Framework provides a structured, multi-stage approach supporting the detailed and interrelated activities required to introduce and utilize automated tools along with security testing best practices with an iterative / spiral development approach. These activities include: test design development, development and execution of test cases, development and management of test data and the test environment, documenting, tracking and closing trouble reports found during testing.