Vaserve, a UK webhosting company says that 100,000 of its customer sites were wiped out in what looks like a zero day attack on HyperVM, a virtualization application they used. The HyperVM was a product of lxlabs.

I checked out the lxlabs product documentation and website and could not find any reference to using a secure development lifecycle. I did find this rather disturbing post to their forum as the first post on a new topic “Security” in April 10, 2009. It was not replied to until yesterday.

http://forum.lxlabs.com/index.php?t=msg&th=11197&start=0&:

Lxadmin/hyperVM has become popular enough that people are SPECIFICALLY targeting these softwares now, and so we are now redoubling our focus on security.

If anybody knows about any vulnerability in hyperVM or lxadmin, please contact lxinfo at lxlabs.com, and we can negotiate a payment if you can demonstrate it clearly.

Of course, after we fix the bug and update the softwares, we will absolutely disclose it publicly too, since we believe in 100% openness, but we need to know about vulnerabilities before it can impact our clients.

Thanks.

This is obviously not a good software security strategy. The owner of the IP is responsible for testing for security flaws. It was obviously too little too late for lxlabs. The industry can learn from this lesson. Don’t wait until your software reaches critical mass and raises the attention of blackhat researchers before you start to think about application security.

The bigger issue and one that Vaserve should be asking itself is why did they place so much trust in software that clearly didn’t have a software security process behind it. Vaserve should have looked for evidence of a 3rd party security review before they accepted the risk of an application that has the potential to bring down their whole company.

Hosting and cloud provider customers need to ask themselves how they vet the providers they use. Have their providers demanded evidence of 3rd party security reviews of the products in their infrastructure stack? Until customers start requiring this evidence these disasters will continue. Evidence of a security review has to start with the end user customer and work its way up the supply chain to hosting/cloud provider and then to the software vendor.

Monster.com recently disclosed yet another major breach that compromised the personal data of over 1.3 million users. This is not unlike the previous breach in August 2007, though the attack vector was likely different. From a notice on their website (emphasis mine):

We recently learned our database was illegally accessed and certain contact and account data were taken, including Monster user IDs and passwords, email addresses, names, phone numbers, and some basic demographic data. The information accessed does not include resumes.

Considering the well-known tendency to use the same password on multiple websites, compounded with the fact that Monster pledged a comprehensive security review after the first breach, it’s just embarrassing that they are still storing passwords in the clear.

So let’s talk about how to properly store passwords for a web application.

Use a one-way cryptographic hash

Don’t store your passwords in the clear! If you do, an attacker just needs to find one SQL Injection vulnerability and he’s got the password for every one of your users. The idea behind using a one-way algorithm is that the hash value can’t be reversed to “decrypt” the password. So how does authentication work? When a user attempts to login, you apply the same one-way algorithm to convert the user-provided password into the hash value, and then compare the two hashes. If they match, then the user-provided password was correct. At no time is the password ever stored in the clear.

Often, developers will hear the advice “use a hash” and interpret that as “run the plaintext password through MD5 or SHA-1 and store the result.” But that only solves part of the problem — the part about using an irreversible algorithm. It doesn’t protect against pre-computation. Let’s say you’ve used SHA-1 to hash your passwords, and your USERS table looks like this in the database:

USER          PASSWORD_HASH
admin         5baa61e4c9b93f3f0682250b6cf8331b7ee68fd8
bob           fbb73ec5afd91d5b503ca11756e33d21a9045d9d
jim           7c6a61c68ef8b9b6b061b28c348bc1ed7921cb53

So if you wanted to obtain the original passwords you’d have to run a dictionary or brute force attack, hashing all possible password options with SHA-1 and comparing the output to the stored hashes. This would take a long time but eventually you’d figure some of them out. But what if you already had a list of all 8-character permutations and their corresponding SHA-1 hashes? Now all you have to do is look up the hashes, rather than computing them on-the-fly. This is the idea behind rainbow tables.

An attacker with a SHA-1 rainbow table covering 8-character alphanumeric combinations would quickly look up those three hashes and obtain the original passwords of “password”, “p4ssword”, and “passw0rd” respectively.

Use a salt

The best defense against pre-computation of raw hashes is salting. To salt a password, you append or prepend a random string of bits to the plaintext password and hash the result. You then store the salt value alongside the hash so that it can be used by the authentication routine. Look in the /etc/shadow file of any modern Unix system and you’ll see something like this:

user1:$1$lKorlp4C$RD5TSM6PaZ6oaWRVUuXT40:13740:0:99999:7:::
user2:$1$qOmA0CUm$I6IdbZDTDl6B6m7s77VPe1:13650:0:99999:7:::
user3:$1$nIEInNo5$PSxcLtvGIJArL8r2AQl74.:13749:0:99999:7:::

Let’s look at the “user1″ entry in the example above, paying attention to the second field which contains a bunch of alphanumeric characters separated by dollar signs. The first token, 1, is a version number, The second token, lKorlp4C, is the salt. The third token, RD5TSM6PaZ6oaWRVUuXT40, is the one-way hash that was calculated using lKorlp4C as the salt.

When the user attempts to login, the system passes the user-provided password along with the stored salt into the hash routine (in this case, md5crypt), and compares the result to the stored hash.

Each bit of salt used doubles the amount of storage and computation required for a pre-computed table. For instance, if we used one bit of salt — either 0 or 1 — the rainbow table would have to account for two variations of every password. Eight bits of salt require 2^8, or 256 variations of every password. Use a sufficiently large salt and pre-computation becomes infeasible. For example, the md5crypt utility uses 48 bits of salt (and for an extra layer of protection, it runs 1000 iterations of MD5 to slow down dictionary attacks).

There are a couple of common mistakes that people make with regard to salting. First, don’t use the same salt every time. If you do, you’re not really increasing the search space because the attacker only has to account for a single salt value. Second, don’t worry about protecting the salt values, they’re not secrets. The added security is derived not from the secrecy of the salt but rather by the amount it increases the resources required for pre-computation.

If you have OpenSSL installed you can play around with various salt mechanisms and see what the output looks like:

$ openssl passwd -h
Usage: passwd [options] [passwords]
where options are
-crypt             standard Unix password algorithm (default)
-1                 MD5-based password algorithm
-apr1              MD5-based password algorithm, Apache variant
-salt string       use provided salt
-in file           read passwords from file
-stdin             read passwords from stdin
-noverify          never verify when reading password from terminal
-quiet             no warnings
-table             format output as table
-reverse           switch table columns

$ openssl passwd -1 password
$1$LH1SwzJI$0ho4XuPVfGlbWIcNuGIap/
$ openssl passwd -1 password
$1$eAUtQOBh$GlvJwVsyb8In5KKkvnR0E0
$ openssl passwd -1 password
$1$PgaSiWTy$ElLh6uy83Y6T4Y70AGmV20

A quick Google search shows that there is a lot of confusion about salting.

But wait, now my password recovery feature won’t work

What’s that? You say your application has one of those “Forgot My Password” features where a user can type in their username and their current password will be sent to the e-mail address on file? Clearly, that requirement depends on passwords being stored either in the clear or using a reversible mechanism such as symmetric encryption.

The answer here is to redesign your password recovery feature. Don’t let an unnecessary requirement force you into poor security practices. If you must e-mail a password, generate a temporary password that’s only valid for a short time period, and require the user to login immediately and select a new password. This obviates the need to retrieve the original, forgotten password.

Why not just use symmetric encryption?

Instead of storing passwords in the clear, you could encrypt them using a symmetric algorithm such as AES and have the application encrypt/decrypt as needed. While this solves the plaintext storage problem, it creates a new problem: key management. Where do you store the key? How often does it change? How many people have access to it? What do you do if/when the key is compromised? And so on. The tradeoff really isn’t worth it for something that’s more elegantly solved with salted hashes.

Layered defenses

While you’re rethinking password storage, it might be a good time to consider other common flubs such as password complexity and brute-force protections.

In conclusion

• Storing passwords in the clear puts your users at unnecessary risk if (when) your application database is compromised
• Use salted hashes instead of storing passwords in a recoverable format
• Password recovery mechanisms can be implemented without needing to obtain the original password
• As with any aspect of security architecture, use layered defenses

Have fun refactoring!

With regard to the recent Patch Tuesday fix, there has been an issue fixed regarding NTLM Relaying, that has been around for more than eight years.

In 2000, I wrote an advisory about NTLM relaying (CVE-2000-0834). The problem turned out to be significantly larger than I originally suggested in the advisory. The attack extended to other NTLM-based authentications on other protocols and allowed general-purpose credential theft via a man-in-the-middle attack.

The SMBRelay tool was published in 2001 by Sir Dystic of Cult Of The Dead Cow, and that really took it to the next level. The protocol completely fell apart. It kicked off a number of other analyses of the NTLM protocol that finally resulted in this patch. Eight years after it’s discovery.

At least they got around to it. Thanks!

Assuming the mailbox hack is not an elaborate ruse, how did they do it?

Almost as bad as the Sprint PCS password reset fiasco that made the news in April, here is the Yahoo Mail password reset screen:

As you can see, you need to know the user’s birthday, country of residence, and postal code. Not difficult information to dig up in Palin’s case, as shown here. After you enter this information correctly, you are asked to type in the alternate e-mail address that’s associated with the account. But they give you hints — so if your alternate e-mail was sarah@alaska.gov, they would show you s****@a*****.gov.

Assuming you guess the alternate e-mail correctly, Yahoo mails a password reset link to that address. So it’s likely that the attacker may have also had to gain access to her alternate e-mail account. Either that, or they exploited a vulnerability in the Yahoo password reset mechanism itself, which seems less likely but not implausible.

So Yahoo itself probably didn’t get hacked, per se, even though there will probably be a lot of FUD in the media about that.

Update 08/18/2008 1:00am EST:

Just found this writeup describing how it transpired: http://pastebin.com/f7fb944c5. Again, not vouching for the authenticity but it does seem plausible, and it’s consistent with my password reset theory. I guess my Yahoo account doesn’t have a secret question defined so I wasn’t presented that option when I tested the reset mechanism earlier today.

Just for fun, here’s the list of non-customizable secret questions Yahoo lets you pick from, as of tonight:

And they sure don’t make it easy for you to update your secret question, do they? (must be logged in to Yahoo for that link to work)

A group of individuals has compromised VP candidate Sarah Palin’s personal email and sent the information to Wikileaks which has posted the information publicly.

http://wikileaks.org/wiki/Sarah_Palin_Yahoo_email_hack_2008

Alternate link (wikilieaks is down): http://cryptome.org/palin-email.zip

Circa midnight Tuesday the 16th of September (EST) Wikileaks’ sources loosely affiliated with the activist group ‘anonymous’ gained access to U.S. Republican Party Vice-presidential candidate Sarah Palin’s Yahoo email account gov.palin@yahoo.com. Governor Palin has come under criticism for using private email accounts to avoid government transparency mechanisms. The zip archive made available by Wikileaks contains screen shots of Palin’s inbox, example emails, address book and two family photos. The list of correspondence, together with the account name, appears to re-enforce the criticism.

Internet security has finally become an issue in presidential politics.

Palin’s use of a Yahoo account has been the subject of recent newspaper articles. The Washington Post published her Yahoo email address, which was likely a precursor to the attack.

Why bother setting up dedicated websites to host malicious content when you can just infect trusted sites like BusinessWeek? This is becoming something of a trend, as evidenced by the mass SQL Injection attacks from a few months ago.

The idea is simple — find SQL Injection vulnerabilities in high-traffic, trusted websites where the site’s content is dynamically fetched from a database (i.e. just about any content-rich site). Then use an automated tool to prepend or append malicious content to that content in the database. When the unsuspecting user visits the page to read an article, they will be treated to a barrage of <script> or other tags fetching content from sites in .ru, .cn, or who knows where else.

The guidance you give to mom and dad, “don’t visit sketchy looking sites in other countries,” is no longer good enough. If BusinessWeek can be compromised, it’s a given that USA Today, CNN, the New York Times, and other establishments are being targeted as well.

For this and similar examples, NoScript would have thwarted the attack because it wouldn’t permit the .js file to be loaded from an off-domain location. But what happens when the attackers start injecting the entire .js payload into the database instead of just a <script> tag? Now the malicious code is coming from the trusted domain, and if I’ve configured NoScript to allow scripts from businessweek.com, I’m out of luck. In fact, I have no idea why the attackers aren’t using this tactic already. Any ideas?

Earlier today, the US District Court dealt a victory to the MBTA hackers and the EFF, lifting the injunction issued on August 9th to prevent the three MIT students from presenting their findings at DEFCON 16. In summary:

The lawsuit claimed that the students’ planned presentation would violate the Computer Fraud and Abuse Act (CFAA) by enabling others to defraud the MBTA of transit fares. A different federal judge, meeting in a special Saturday session, ordered the trio not to disclose for ten days any information that could be used by others to get free subway rides.

“The judge today correctly found that it was unlikely that the CFAA would apply to security researchers giving an academic talk,” said EFF Staff Attorney Marcia Hofmann. “A presentation at a security conference is not some sort of computer intrusion. It’s protected speech and vital to the free flow of information about computer security vulnerabilities. Silencing researchers does not improve security — the vulnerability was there before the students discovered it and would remain in place regardless of whether the students publicly discussed it or not.”

This sets a good precedent for future cases, and perhaps next time a similar situation arises, a judge will not be so quick to issue a gag order. It’s not a happy ending yet though, as the original lawsuit is still in effect.

As Chris Wysopal pointed out last week, the MBTA’s ire is misdirected. Rather than suing the vendor who sold them the defective system, they sued and attempted to silence the students who discovered the weakness. This is 2008, not 1988 — did they honestly think a gag order would prevent the information from reaching the general public? The DEFCON presentation was already available on the Intertubes prior to the injunction being issued, and the MBTA attorneys included a copy of the confidential whitepaper with their filing, thereby making it public.

I guess you wouldn’t expect that a transit authority would have paid any attention to the Ciscogate fiasco from a few years ago. That presentation never got out either, did it? All that taxpayer money the MBTA spent on ridiculous lawsuits and restraining orders could have been put toward fixing the security flaws. What a concept.

By now, you probably know that details of the DNS vulnerability have leaked. Halvar Flake speculated on DailyDave and the momentum built from there, despite the fact that his guess was short on a few key details. I don’t need to rehash the full technical details here; by now, they are easy enough to find with a couple Google searches. When Slashdot picks up the story, it’s hardly a secret any more.

What’s more interesting to me, now that I’ve digested the big secret, is how this whole situation has played out in the security community.

The security community has been polarized for the past two weeks, not so much over the technical details being withheld, but about Dan’s plea that people not speculate about the vulnerability. As many pointed out, the “bad guys” won’t stop trying to figure it out just because the “good guys” keep quiet. To be honest, my own lack of public speculation wasn’t because I agreed with the philosophy; I just wasn’t smart enough to figure out the vulnerability myself.

People implied — or stated outright — that Dan just didn’t want anyone stealing his thunder. Considering the timing of the release and the subsequent BlackHat talk, it’s obvious why such accusations were made. Personally, I think it’s a little of each. I believe the coordinated patch effort was undertaken with the best of intentions, but I also think Dan relished some of the glory and media attention as well. It’s hard to blame him for that; if you were in his shoes, wouldn’t you want some recognition too?

By many accounts, dealing with the DNS vulnerability from the operational side has been an exercise in frustration. Plenty of IT people wanted to patch but couldn’t get approval without being able to justify the operational risk. “Because Dan said so” is apparently not a convincing enough argument. Some wondered why the people who were responsible for creating the problem should be blindly trusted to implement an appropriate fix?

Ultimately, vulnerability disclosure is a minefield. No matter how you choose to disclose, somebody will always disagree.

P.S. If you didn’t figure out the title of the post by now, Nate was one of the unlucky few to draw the same timeslot at BlackHat as Dan Kaminsky.

A co-worker passed along this snapshot taken at the Karsten Nohl, Jake Appelbaum, and Dino Dai Zovi talk at HOPE this past weekend. The context, of course, is that the overzealous Debian developer who accidentally crippled OpenSSL back in 2006 said he did so because valgrind reported uninitialized memory use. Click through for the full-size version.

So automated software review is dangerous now? Perhaps that bullet should read “modifying code you don’t understand is dangerous.”

The security community is cynical. So much so, that most of the chatter that’s taken place over the past 24-36 hours has suggested that Kaminsky’s DNS vulnerability was little more than a publicity stunt and that his BlackHat presentation would be an over-hyped rehash of prior art. Granted, one has to suspend disbelief to even consider that something monumental would be discovered in DNS — that’s the protocol itself — but hell, it’s always nice to give a guy the benefit of the doubt.

Faced with nearly a month of criticism and questioning, and understanding the persuasive power of a technical peer review, Dan decided to expand the inner circle, so to speak. Rich Mogull arranged a phone call with Tom Ptacek and Dino Dai Zovi so that Dan could spill the beans and let them decide for themselves whether it was spin or substance. Turns out there was substance.

Now we sit around and wait until August 6th to cram into a ballroom with a thousand sweaty conference-goers to hear the juicy details. And Dan’s presentations are usually packed to the brim even when he’s not announcing anything.

In the meantime… how about patching those servers?