Here’s the rest of my list:

10:00-11:00 FX, Developments in Cisco IOS Forensics.

11:15-12:30 Oliver Friedrichs, Threats to the 2008 Presidential Election (and more).

13:45-15:00 Option 1: Scott Stender, Concurrency Attacks in Web Applications. Option 2: Travis Goodspeed, Side-channel Timing Attacks on MSP430 Microcontroller Firmware.

15:15-16:30 Option 1: Alexander Sotirov and Mark Dowd, How To Impress Girls With Browser Memory Protection Bypasses. Option 2: Karsten Nohl, Mifare - Little Security, Despite Obscurity. This is one of the toughest time slots as you also have McFeters/Carter/Heasman and Grossman/Evans in the lineup. Choices, choices.

16:45-18:00 Option 1: Bruce Dang, Methods for Understanding Targeted Attacks with Office Documents. Option 2: Christopher Tarnovsky, Inducing Momentary Faults Within Secure Smartcards/Microcontrollers.

Lots of intriguing hardware talks on Day 2. A lot of it is probably over my head and my first options are more applicable to my day job. There might have to be some room hopping.

I fly out to Vegas tonight — see you all there!

Last week, Ben Worthen of the Wall Street Journal had a conversation with Howard Schmidt about the vulnerabilities in purchased software while Howard was waiting on line to have his iPhone upgraded.

Howard Schmidt, who was once the CSO of Microsoft, knows a thing or two about vendors shipping insecure software. He offers this advice relating to his iPhone, “Just because a piece of software was distributed through Apple’s App Store, don’t assume that it is vulnerability free.” I think that sums up the problem pretty well. Customers assume the software they are getting is vulnerability free until it is proved otherwise.

If it’s distributed by the Apple Store it is coming from a trusted brand. “It must be secure”, many think. The same thinking is used by people who install social networking applets and give them access to their personal data. Someone, somewhere is taking care of the software security so I don’t have to. It must be the platform provider, the store, some industry body, my antivirus provider, or maybe even the government.

You can see how this thinking pervades the consumer space because there are regulatory bodies governing all other aspects of safety and security in our personal lives. I’m safe in a plane or car because the government is looking out for me with standards and testing requirements. I am safe in the mall parking lot because the men in the white SUV are patrolling.

This thinking also pervaded the b2b space. I talk to companies which are outsourcing critical applications to offshore development companies and they assume that security testing is taking place as part of the development process. I ask them if they have made security quality part of the requirements of the project and they say no. Then I ask them what evidence does the offshore developer provide to demonstrate they have a certain level of security quality in the software they are producing and they tell me they have never asked.

I can tell you what would happen if they did ask because I have also spoken with the offshore developers. They have no evidence. Their concern is getting the software functionality done on time and on budget. They consider fixing security vulnerabilities, once discovered, rework which the customer pays for. So not only are they not looking for vulnerabilities and relying on the customer to find them, they are charging the customer to fix the problems. The customer has to this date accepted this model.

The same goes for commercial off the shelf software and open source. Surely the developers writing the software are trained in secure software engineering. Surely commercial software companies are using 3rd parties to test their software just like the banks have the big 4 audit their accounting or auto manufacturers submit to testing by the NHTSA. And of course open source has “many eyes” reviewing the code for security defects and informing the developers. The customer has accepted a model where this is almost never true.

But times are changing and it is partially due to the availability of software that can automate the process of looking for security vulnerabilities. David Rice, the author of “Geekanomics: The Real Cost of Insecure Software” was interviewed recently by Drazin Drazic his Beast or Buddha blog. He said the trend is toward a future of secure software and automated security analysis is one of the sparks:

BorB: I recently wrote in a post that little is changing. We are not learning from the lessons of the past. There are few, if any new technologies that exist today, that we have great faith and trust in as being secure now, and expecting them to continue to be secure in the future. Any solutions to even basic security issues need a starting point and a significant change to current thinking, and even then, it will takes years to see the impacts of this. What are your thoughts on this? Are we seeing anything at present to make us more confident of the future?

DR: It is true that it takes years to see the positive impacts of a change of mindset. And we are in the unfortunate position of repeating many old lessons.

At base, human history is a collection of exhaustive, expensive, and protracted engagements; only the relentless survive and have a chance at succeeding (notice no guarantee here). Confronting some of our most complex problems like highway safety, nuclear proliferation, or insecure software is painful, difficult, complicated, and troublesome. Human endeavors of any significance are like this. But we must do it. The inertia of culture and status quo is difficult to overcome, but overcome it we can; otherwise, we would not have the better parts of the world we enjoy today.

I believe the technology space is no different. We are just a little dazed and bewildered by all the changes technology has introduced so quickly and on such a grand scale. For every change we react to, another two or three rapidly appear.

I do see sparks of hope emerging. In the United States some members of government are beginning to understand the problem and are willing to start discussing how to approach insecure software from a policy perspective. On the technology front, companies like Ounce, Fortify, and Veracode are beginning to give software buyers an automated method of evaluating assurance levels of software. While not complete in and of themselves, these solutions are, as I stated, “sparks” that can help us progress down paths that were once not easily open to us.

As for the larger issue of cyber security, which software assurance is only a part of, society has a lot of adjusting to do. The Internet is a new environment for many still, and many more to come. There is a learning curve that must be confronted. It took the United States almost 80 years to develop the highway system we know and enjoy today. Nearly $400 billion was spent on this endeavor with hundreds of thousands of lives lost. As this shows, learning how to govern and navigate a new environment is expensive. Failing to learn even more so.

Independent, automated, and repeatable software security testing is an essential component of a safe and secure online environment. Without it we are stuck with the assumption of vendors perfoming software security as our imaginary security blanket that allows us to operate in the current online world.

Well, it’s almost BlackHat time. Here are my picks so far for Day 1. As you can see, I still haven’t narrowed it down completely.

11:15-12:30 Option 1: Dan Kaminsky, “DNS Goodness”. On one hand, the DNS vulnerability is already public; on the other hand, the talk will probably still be interesting even if the 0day hype is missing. Option 2: Nate Lawson, “Highway to Hell: Hacking Toll Systems”. My formal education and early work was in Electrical Engineering, so I’m always interested in hardware talks. I haven’t touched a soldering iron in years so I have to live vicariously through people like Nate.

13:45-15:00 Option 1: Chris Hoff, “The Four Horsemen of the Virtualization Security Apocalypse”. I haven’t been paying enough attention to virtualization security and I think this talk will be quite informative. Option 2: Danny Quist and Colin Ames, “Temporal Reverse Engineering”. Sounds like an interesting approach.

15:15-16:30 Option 1: Hovav Shacham, “Return-Oriented Programming: Exploits Without Code Injection”. The topic sounds pretty straightforward conceptually but it will be interesting to see the implementation. Option 2: Tom Stracener and Robert Hansen, “Xploiting Google Gadgets: Gmalware and Beyond”. Not expecting any huge revelations on this one but it’s likely to be entertaining.

18:00-19:00 The Pwnie Awards. Turnout last year was kind of slim, but I bet the room will be full this year as it’s been publicized more.

Day 2 picks coming soon!

By now, you probably know that details of the DNS vulnerability have leaked. Halvar Flake speculated on DailyDave and the momentum built from there, despite the fact that his guess was short on a few key details. I don’t need to rehash the full technical details here; by now, they are easy enough to find with a couple Google searches. When Slashdot picks up the story, it’s hardly a secret any more.

What’s more interesting to me, now that I’ve digested the big secret, is how this whole situation has played out in the security community.

The security community has been polarized for the past two weeks, not so much over the technical details being withheld, but about Dan’s plea that people not speculate about the vulnerability. As many pointed out, the “bad guys” won’t stop trying to figure it out just because the “good guys” keep quiet. To be honest, my own lack of public speculation wasn’t because I agreed with the philosophy; I just wasn’t smart enough to figure out the vulnerability myself.

People implied — or stated outright — that Dan just didn’t want anyone stealing his thunder. Considering the timing of the release and the subsequent BlackHat talk, it’s obvious why such accusations were made. Personally, I think it’s a little of each. I believe the coordinated patch effort was undertaken with the best of intentions, but I also think Dan relished some of the glory and media attention as well. It’s hard to blame him for that; if you were in his shoes, wouldn’t you want some recognition too?

By many accounts, dealing with the DNS vulnerability from the operational side has been an exercise in frustration. Plenty of IT people wanted to patch but couldn’t get approval without being able to justify the operational risk. “Because Dan said so” is apparently not a convincing enough argument. Some wondered why the people who were responsible for creating the problem should be blindly trusted to implement an appropriate fix?

Ultimately, vulnerability disclosure is a minefield. No matter how you choose to disclose, somebody will always disagree.

P.S. If you didn’t figure out the title of the post by now, Nate was one of the unlucky few to draw the same timeslot at BlackHat as Dan Kaminsky.

A co-worker passed along this snapshot taken at the Karsten Nohl, Jake Appelbaum, and Dino Dai Zovi talk at HOPE this past weekend. The context, of course, is that the overzealous Debian developer who accidentally crippled OpenSSL back in 2006 said he did so because valgrind reported uninitialized memory use. Click through for the full-size version.

So automated software review is dangerous now? Perhaps that bullet should read “modifying code you don’t understand is dangerous.”

Despite what various commenters around the blogosphere think (I’ve read a few but can’t find the links now), Dan Kaminsky’s online “Check My Dns” utility doesn’t:

• Poison anybody’s DNS cache
• Expose how the actual exploit works

What it does is check whether your ISP’s DNS server is patched. Plain and simple. It looks for one thing — source port randomization. This does not give away the exploit, it checks for the existence of the sledgehammer fix that prevents the exploit from working.

More specifically, there’s some Javascript code that generates a random hex string which is used to create a URL, e.g. http://6313d97e498e.toorrr.com. Your OS then does a DNS lookup for that unique hostname. Your ISP’s DNS server asks toorrr.com’s DNS server (a server Dan controls) to resolve that funky DNS name to an IP address. It sends a few packets in the process. Dan’s server makes a note of the source port of each request and sends back the webserver’s IP address to your DNS server, which sends it back to you.

Now that you have the IP address, your browser can fetch the results page. The web page is generated dynamically by parsing the hex string out of the URL you requested, using Ajax to fetch the relevant port and TXID data stored on Dan’s server, and printing out a “safe” or “vulnerable” message such as:

Your name server, at 71.243.0.38, appears to be safe.
Requests seen for 6313d97e498e.toorrr.com:

71.243.0.38:45298 TXID=13926
71.243.0.38:45310 TXID=25412
71.243.0.38:45338 TXID=30829
71.243.0.38:45332 TXID=13934
71.243.0.38:45321 TXID=2701

That’s all. Nothing tricky. This particular DNS server is deemed safe because the source port varies from one request to the next.

Come to think of it, those source ports don’t really look that random, do they? For anybody “in the know”, is that amount of randomness sufficient to protect against the attack?

The security community is cynical. So much so, that most of the chatter that’s taken place over the past 24-36 hours has suggested that Kaminsky’s DNS vulnerability was little more than a publicity stunt and that his BlackHat presentation would be an over-hyped rehash of prior art. Granted, one has to suspend disbelief to even consider that something monumental would be discovered in DNS — that’s the protocol itself — but hell, it’s always nice to give a guy the benefit of the doubt.

Faced with nearly a month of criticism and questioning, and understanding the persuasive power of a technical peer review, Dan decided to expand the inner circle, so to speak. Rich Mogull arranged a phone call with Tom Ptacek and Dino Dai Zovi so that Dan could spill the beans and let them decide for themselves whether it was spin or substance. Turns out there was substance.

Now we sit around and wait until August 6th to cram into a ballroom with a thousand sweaty conference-goers to hear the juicy details. And Dan’s presentations are usually packed to the brim even when he’s not announcing anything.

In the meantime… how about patching those servers?

Rich Mogull’s executive overview of Dan Kaminsky’s latest DNS vulnerability fluffed a few feathers yesterday:

The good news is that due to the nature of this problem, it is extremely difficult to determine the vulnerability merely by analyzing the patches; a common technique malicious individuals use to figure out security weaknesses.

The typical response I heard was “what do you mean, it can’t be reverse engineered? I’ll just look at the diffs!”

In hindsight, after examining the BIND diffs (yes, I did it too) and discussing with colleagues, all most people saw was UDP source port randomization and a better PRNG for generating the transaction ID, the latter of which would appear to be related to Amit Klein’s cache poisoning attack from about a year ago.

What Rich was really saying is that you can reverse engineer the patch until you’re blue in the face, but that won’t reveal the specifics of the vulnerability.

Dan’s blog post this morning appeared to confirm that interpretation:

DJB was right. All those years ago, Dan J. Bernstein was right: Source Port Randomization should be standard on every name server in production use.

There is a fantastic quote that guides a lot of the work I do: Luck is the residue of design. Dan Bernstein is a notably lucky programmer, and that’s no accident. The professor lives and breathes systems engineering in a way that my hackish code aspires to one day experience. DJB got “lucky” here — he ended up defending himself against an attack he almost certainly never encountered.

Such is the mark of excellent design. Excellent design protects you against things you don’t have any information about. And so we are deploying this excellent design to provide no information.

To translate the fix strategy into a more familiar domain, imagine large chunks of Windows RPC went from Anonymous to Authenticated User only, or even all the way to Admin Only. Or wait, just remember Windows XPSP2 :) This is a sledgehammer, by design. It cuts off attack surface, without necessarily saying why. Astonishingly subtle bugs can be easily hidden, or even rendered irrelevant, by a suitably blunt fix.

Nate McFeters appears to think that Tom Ptacek has figured it out. I’m going to go out on a limb and say that Tom didn’t figure anything out yet but still wanted to write a pithy blog post. I think that if Tom had figured it out, he would have written it down privately and posted the SHA-1 hash, as is the trendy thing to do these days.

Speculation aside, the title of Tom’s blog entry, Dan Kaminsky could have made hundreds of thousands of dollars with this DNS flaw!, does make an important point — Dan didn’t sell the details to ZDI, he used his influence and reputation to coordinate a massive vendor patch effort. That’s an admirable move.

I’m finally getting around to finishing my post on minimizing attack surfaces. Here’s Part 1, in case you missed it.

First, a quick clarification. I noticed that some of the readers who commented on that first post wanted to talk about improving security through the use of various development methodologies or coding frameworks. Those are interesting tangents (and ones that I may write about in the future), but my intention with this post is to discuss a very specific problem related to how people integrate third-party code — that is, the stuff you import or link in but didn’t write yourself.

As I mentioned previously, developers have a tendency to “bolt on” third-party components to applications without understanding the security implications. Often, these components are glossed over or ignored completely during threat modeling discussions. I attempted to illustrate this with my fictitious WhizBang library example in Part 1.

When integrating a third-party component, developers familiarize themselves with the API but generally don’t care how it’s implemented. Granted, that’s how an API is supposed to work; you don’t have to futz around with code beyond the API boundary, and you can blissfully ignore parts of the library that you don’t need. In past consulting gigs, I’ve sat in threat modeling discussions where nobody knew whether a particular library generated network traffic. “We just use the API,” they say. The fact that it works is good enough; nobody seems to care how it works.

That mindset is ideal for rapid development but problematic for security. Failing to understand the complete application, as opposed to just the part you wrote, prevents you from accurately assessing its security posture.

It’s also no coincidence that web app pen testers love third-party components — we get excited when we see “bolted on” interfaces, because we know that developers tend to leave extraneous functionality exposed. The resulting findings usually generate reactions such as “I didn’t even know that servlet had an upload function.”

An Example

Here’s a close-to-home example related to my post about DWR 2.0.5 from the other day. DWR is an Ajax framework that has a variety of operating modes. In-house, we use a subset of DWR’s full functionality — specifically, we interact with it using the “plaincall” method only, so we made sure that the features we didn’t need were disabled via the configuration file. As it turned out, there were vulnerable code paths prior to the “do you have this thing disabled” check. In hindsight, if we had taken more time to understand the exposed interfaces, we could have reduced the attack surface by filtering out unneeded request patterns before they even touched the third-party code.

But wait, you say. What about maintainability? If I whitelist using a point-in-time application profile, doesn’t this create the same maintenance headache as the reviled WAF? It doesn’t have to. Certainly, one option would be to whitelist each and every unique URL that references the DWR framework, e.g.

/dwr/call/plaincall/myMethod1
/dwr/call/plaincall/myMethod2
/dwr/call/plaincall/myMethod3

But then you’d have to update the whitelist every time you added or removed functionality from your application. Also, don’t lose sight of the security goal, which is to minimize the amount of exposed third-party code. If I add or remove URLs that list, provided they are still using the “plaincall” method, I’m hitting the same DWR dispatcher every time. So I’ve increased maintenance cost without any security benefit.

A better option is to simply tighten the URL pattern a bit in the J2EE container. Here’s the default configuration:

<servlet-mapping>
  <servlet-name>dwr-invoker</servlet-name>
  <url-pattern>/dwr/*</url-pattern>
</servlet-mapping>

Now, instead of allowing every URL starting with /dwr/ to be processed by the DWR library, you could be a little more restrictive:

<servlet-mapping>
  <servlet-name>dwr-invoker</servlet-name>
  <url-pattern>/dwr/call/plaincall/*</url-pattern>
</servlet-mapping>

In this configuration, you don’t have to worry about /dwr/call/someothercodepath any more. There is less third-party code exposed, thereby reducing the overall attack surface of the application. (NB: DWR also serves up a couple of Javascript files, so those URL patterns will have to be whitelisted too)

A Logical Extension

Even if you’re not a developer, you should still be thinking about attack surfaces. People download and install blogging platforms such as WordPress, Movable Type, etc. all the time, but how many take additional steps to harden their installations? The concept is the same as the OS hardening analogy I brought up at the very beginning of this discussion.

Similarly, people install third-party WordPress plugins or Joomla components without considering that most of them are written by some random programmer who is a whiz with the plugin API but knows nothing about security?

At the risk of sounding trite, always remember that security is only as strong as the weakest link.

Popular Mechanics recently published an article about the NSA Red Team, which caught my interest, having been a part of that organization for a short stint back in early 2000. The article does a decent job of describing the Red Team’s charter, which is essentially to attack DOD targets in an attempt to simulate real adversaries, not unlike a consultant running a pen test against a corporation. The rules of engagement are similar to most pen tests: don’t DoS the target, don’t install malware, generally be non-destructive.

Disappointingly, the author sprinkles the usual super-secret uber-hacker spin throughout the article to make the Red Team seem mysterious and exclusive, with untouchable talent. It’s a little misleading. For starters, there’s the predictable question about success rates:

I’d heard from one of the Department of Defense clients who had previously worked with the NSA red team that OWNSAVAOG and his team had a success rate of close to 100 percent. “We don’t keep statistics on that,” OWNSAVAOG insisted when I pressed him on an internal measuring stick.

This is one of those statements that is difficult for the average reader to interpret. It’s intended to make the team sound like a crack squad of hackers, but in reality it’s the same statistic that every security consultancy cites during sales calls. The truth is, there’s a lot of wiggle room on what is considered “getting in” to the target. For example, some would say that brute forcing an FTP server and downloading some FOUO (For Official Use Only) documents constitutes penetrating the target. Others would disagree.

How about personnel? I thought this was an englightening and accurate statement from the unnamed NSA source:

And like any good geek at a desk talking to a guy with a really cool job, I wondered just where the NSA finds the members of its superhacker squad. “The bulk is military personnel, civilian government employees and a small cadre of contractors,” OWNSAVAOG says. The military guys mainly conduct the ops (the actual breaking and entering stuff), while the civilians and contractors mainly write code to support their endeavors. For those of you looking for a gig in the ultrasecret world of red teaming, this top hacker says the ideal profile is someone with “technical skills, an adversarial mind-set, perseverance and imagination.”

He basically admits that the team consists mostly of people who “run the tools” and only a handful that actually write the tools or do anything cutting-edge. It shouldn’t be that surprising; just as in any large consulting organization, you have some people who run scanners/tools and aren’t expected to be terribly analytical. While the Red Team almost certainly has some superstars, on the whole it is similar in both skillset and composition to a typical consultancy or enterprise security team.

In terms of attracting and retaining top talent, the Red Team faces the same challenges as the rest of the information security industry, with the built-in disadvantage of the government pay scale. If that wasn’t bad enough, they also have to compete with themselves (i.e. the rest of the NSA) for already scarce resources. Given these challenges, how could one realistically expect the Red Team to be as advanced as the article portrays?

Finally, let’s dispel the “super-secret” notion — unless things have changed significantly, the majority of Red Team operations are unclassified. Granted, detailed information is guarded, but you can find reports summarizing past operations if you dig around a bit. One would expect that an operation intended to be truly secretive would never make its way into Google search results.

I want to conclude by saying that this post is not intended to cast the Red Team itself in a negative light. I enjoyed my time there and had the opportunity to work with some smart people. The Red Team’s goals are worthy and noble; clearly, state-sponsored cyberterrorism is a growing concern and as a country we should be as prepared as possible. But realize that we have a long way to go.