What’s interesting to me about this breach is that Zappos is renowned for their customer service, so watching how they communicate in the coming days and weeks should be an interesting case study. A few notable points so far:

• Tony Hsieh, the CEO, posted a copy of the internal email that they had sent to all their employees. And tweeted about it. The only difference from the customer-facing email was that it stated the number of records affected. But it’s unusual for a company to share internal communication like this.
• Zappos expired everybody’s password, forcing customers to follow the password reset workflow before regaining access to their account. Usually a company will urge you to change your password but won’t force you to do so. This was a good move on their part. The servers seemed very overloaded though; around 9pm last night it took me a few minutes (and a couple of server timeouts) to successfully reset my password.
• Around the same time, Zappos disabled international access to their website, meaning that anybody outside the US couldn’t reset their password as instructed in the email. This seemed a bit odd. As I am writing this post, the site is still unavailable to international customers. This has understandably frustrated some customers.
• In the customer-facing email, Zappos notes that credit card numbers were not affected, but “cryptographically scrambled” passwords were. This is where I believe they ought to be more forthcoming. What does “cryptographically scrambled” entail? An unsalted MD5 hash, Stratfor style? Salted hashes? Symmetric encryption? A homegrown algorithm? Something stronger like bcrypt or scrypt? This detail is critical, because it indicates how easy it will be for attackers to recover the original passwords from the affected customers and try to use them on other sites like Gmail, Facebook, Twitter, and others. Customers might be more likely to change their password on other websites if they understood the relative risk.
• The email does not disclose how long customer data was exposed prior to the breach notification. This is an important detail that was omitted.
• Zappos has been actively engaging with customers on their @Zappos_Service Twitter account. In fact, last night when I posed a question to the CEO’s Twitter alias, @Zappos_Service responded 4 minutes later. They didn’t have an answer, but they responded.
• They turned off their phone system because they felt responding via email would be more efficient (and their phone system couldn’t handle the volume anyway). Still, can you imagine a “typical” company doing this? It seems simultaneously crazy and brilliant.
• It takes a long time to send 24+ million emails. I received mine last night at 8:34pm and 9:03pm, but a colleague here at Veracode mentioned he didn’t get his until this morning. So assuming they’re going out alphabetically by e-mail address, that’s how long it took to get from “c” to “r”.
• Since both Zappos.com and 6pm.com were affected, it’s possible that they shared a single database. There are a bunch of scenarios though. It could be a vulnerability in application code shared by both sites. It could have also been an insider attack, but the fact that credit card numbers were not compromised suggests to me that the attack was external.

For me, the two things to watch for now are how quickly they restore international access and whether or not they disclose how passwords were stored and what “cryptographically scrambled” means in that context. Security breaches happen to the best of companies and these days what differentiates you is how you respond. So far I believe Zappos is on the right track.

(Incidentally, Tony Hsieh’s book, Delivering Happiness, is a fantastic read. I have a lot of respect for how this company operates, and not just because my shoes arrive overnight.)

Veracode Security Guides

Data Security Resources

Just before the holidays, we detected a cross-site scripting (XSS) vulnerability while running a web application scan for one of our customers. Nothing special about that; we detect thousands of these things every week. But as we discussed this particular finding, we noticed that the layout of the website looked… familiar. As it turned out, the discussion forum where we found the XSS was a SaaS-based product called Lithium.

From Lithium’s website: “The world’s most innovative companies such as AT&T, Barnes & Noble, Best Buy, Sephora, Univision, Home Depot, and HP use Lithium to engage their customers in breathtaking new ways (literally, breathtaking).” Run this Google search and you’ll find a bunch of Fortune 500 companies using their software.

So now, instead of one XSS, we have hundreds. It’s not just our customer who is impacted. Suddenly it’s kind of a big deal.

Here’s how everything played out. We’ll do this timeline CORE Security style (all times EST).

• 2011-12-15 5:29 PM. We fire off a quick email to the address listed on Lithium’s Security page.
• 2011-12-15 6:12 PM. We receive a response from Misha Logvinov, Lithium’s CIO.
• 2011-12-15 6:23 PM. We encrypt and send over the vulnerability details to Lithium.
• 2011-12-15 11:40 PM. Lithium reports they have a patch ready to go and will update in the morning.
• 2011-12-16 2:30 PM. We do a little poking around and it seems the vulnerability is patched for some domains but not others. We email for a status check.
• 2011-12-16 2:37 PM. Lithium confirms that the patch is in the process of being rolled out and will be completed by close of business.
• 2011-12-16 COB-ish. We’re not seeing any more vulnerable instances.

Anybody who has reported vulnerabilities before can appreciate how unusual it is for a vendor to respond this quickly. Everything was accomplished in under 24 hours! That is practically unheard of.

From a “big picture” perspective, this whole situation illustrates some important application security themes:

• It’s a canonical example of software supply chain risk. A single XSS vulnerability simultaneously affected hundreds, maybe thousands of customers. No matter how securely these companies developed software internally, they were still exposed to vulnerabilities in third-party software.
• It emphasizes the ecosystem effect of vendor security assessments. One Lithium customer did an analysis of third-party code they were operating. A defect was found, and the vendor fixed it. Now all Lithium customers benefit, without having to lift a finger! Imagine if all companies assessed their third-party code and insisted on fixes from their suppliers.
• It shows that SaaS can have huge security benefits. Imagine if Lithium had been deployed as an on-premise product at each customer sites, requiring each customer to download and install a fix themselves. Some companies would probably never get around to patching their servers. The flip side is if the SaaS company dragged their feet — or simply refused — to patch the software, leaving the customer without a viable mitigation.
• It demonstrates that application security response can be done right. Lithium engaged quickly, took the vulnerability report seriously, and wasted no time in fixing the problem. It’s not uncommon for some vendors to take months.

Increasingly, we’re seeing our customers rethink how they vet the software they purchase or license from third-party suppliers. I hope success stories like these become commonplace as we start holding software suppliers — both SaaS and on-premise — accountable for security, not just functionality.

Veracode Security Guides

Data Security Resources

Backdoors in industrial control systems

These backdoor revelations in industrial control equipment are becoming frequent. Earlier this year Dillion Beresford found similar backdoor vulnerabilities in Siemens equipment.

We find these types vulnerabilities fairly often when we scan vendor code on behalf of our customers at Veracode. Our recent State of Software Security Report vol. 4 detailed the findings. We didn’t find these backdoors in internally developed, outsourced, or open source applications. We did find backdoors in 3% of software vendor developed code.

This chart above is the result of our static and dynamic analysis of thousands of different applications over the preceding 18 month period.

Vendors add this backdoor code because it lowers their support costs. Unfortunately it is at the expense of the customer’s risk. It is easier for a vendor support technician to remotely diagnose a problem if they know a “support” password to your system or if there is a debugging interface exposed to the network. No need to fly on site or communicate time consuming “remote hands” commands to a local IT employee.

We have seen an uptick in customers performing 3rd party scans on the software they are purchasing. A few years ago it was only our financial services customers that were concerned about backdoors and vulnerabilities in the code they were purchasing. Now we are seeing a much broader range of industry verticals.

The chart above shows we have 8 different industry types including: aerospace & defense and oil & gas, scanning 3rd party code. We are still not seeing industrial control equipment but with the news this year I think it is only a matter of time. 3rd party analysis will grow as operators of code continue the trend to hold vendors accountable.

Backdoor testing should always include static code scanning. How can you find a static password or cryptography key without it? Ideally this is done on the product binary. Vendors are loath to give up source code, even to a 3rd party, and even if they do they might not give you the exact source code or all of the source code. Binary scanning and backdoor testing go hand in hand so Veracode has done research on the subject of backdoor and implemented as much as was practical in our binary static analysis. For further reading on testing apps for backdoors see our “Static Detection of Application Backdoors” paper which was presented at Black Hat Las Vegas.

The recurring theme in her manifesto is the notion that certain software suppliers are “too big to test”. It’s fine for the little guys, but not the 800-pound gorillas. Instead, software purchasers should blindly trust companies with security teams and assurance processes to produce secure code. If only it were that simple. In fact, according to our semi-annual State of Software Security Report, there’s negligible variation in security quality across software suppliers regardless of company size.

We’re both flattered and amused that Ms. Davidson believes our company alone “created a market” for testing the software supply chain. On the contrary, the market has created itself. Take a look through the noteworthy breaches from the past 12-24 months; software vulnerabilities have been the culprit in nearly every case. CISOs are waking up to the stark realization that all software — internally or externally produced — introduces risk into their organizations. In this day and age, wise companies harbor a healthy suspicion of their software vendors. Oracle can choose to do security testing in-house, but a company that’s “running their entire business” on Oracle’s software has a right to request unbiased evidence that the testing process is working.

That being said, Oracle is hardly the poster child for security process. Within the security community, they are notorious for shipping insecure products. Their laughable “Unbreakable” marketing campaign was famously debunked by security expert David Litchfield, who uncovered several critical (and easily avoidable) vulnerabilities within a matter of weeks. They’ve also earned a reputation for glacial response times and sloppy patches. No company can be expected to build perfectly secure software, but it’s pretty obvious why external validation is needed to complement in-house process — one need look no further than ZDI for evidence. Even Ms. Davidson’s own example illustrates how an outsourced service provider “HuiMaika’i” detected multiple vulnerabilities that weren’t discovered by Oracle’s internal team.

Perhaps the most shocking admission about Oracle’s security program is their interpretation of the “need to know” principle. Ms. Davidson asserts that she doesn’t need access to bug databases. This is a classic liability avoidance move and one that we’ve witnessed in other organizations as well. Creating barriers to vulnerability information facilitates a culture in which the executive has plausible deniability of critical bugs and can simply look the other way if a ship deadline is looming or if the auditors pay a visit. CISOs should be clamoring for as much data as they can get their hands on, not eschewing it.

Finally, Ms. Davidson seemed offended that a tenured university professor would suggest licensing software developers to create a system of accountability. Ironically, only a few years ago, she sent a letter to top universities pressuring them to incorporate secure coding guidelines such as the SANS coding certification into their curriculums. She told them, “We will start making our purchasing decisions, if you will, based on that.” Apparently, it’s OK for Oracle to flex their muscle when “buying” (i.e. hiring) from universities, but it’s not OK for Oracle’s customers to hold them to similar standards? It certainly sounds like Oracle has been feeling the pressure lately.

There are third-party tests and assessments for perhaps every important purchase in business or in our personal lives. Companies hire law firms and specialists when they make acquisitions. People look to safety and quality tests from trusted sources before they buy everything from baby strollers to cars. You wouldn’t think of buying a home without a home inspection. In each case, the cost of the independent test must be commensurate with the purchase price and the risk. Look at the typical due dilligence around home purchase. It doesn’t always make sense to pay an engineering firm thousands of dollars for a structural analysis, but it does make sense to hire a home inspector for a few hundred dollars, who in a few hours can uncover termites or a leaking roof. These are problems that must be fixed, and because the testing cost is so low it would be negligent not to do it. Most of the major software vendors have participated in third-party testing either as part of their SDLC, to vet code they were acquiring or licensing, or as part of one of their customers’ procurement process.

Veracode has never claimed that binary SAST provides complete software assurance. From the beginning, we have recommended multiple testing methods to detect vulnerabilities that static automation can’t. In fact, it’s impossible to receive our top ratings without a clean bill of health from a manual penetration test. Each layer of testing, while imperfect on its own, uncovers problems that must be corrected.

Outsourcing is not a dirty word. Many companies outsource development for entire products or components of them. Companies also outsource testing and training. The multi-billion dollar IV&V market grew out of this need — it’s simply good business. The goal is shipping secure code, not making a feel-good proclamation that your team can handle a modern development challenge with no outside help. While Oracle can be proud that they have tamed a source code tool and lived to tell the tale, other companies are securing their code faster and cheaper with the help of outsourcing. Even Veracode customers haven’t fully outsourced security; many of them have in-house security expertise and are just employing a service to make their security processes more robust. They are still full participants in the process, making decisions around how/when to remediate, how much to invest, etc. Veracode acts as an application security partner, providing customers valuable intelligence gleaned from the software ecosystem. Just as Google gets smarter with every search that it does, Veracode gets smarter with every scan we do.

At least we can rest easy knowing that Oracle would never hire lobbyists to promote an agenda. That’s a relief!

Veracode Security Solutions

Security Alternatives

Security Threat Guides

At CanSecWest security conference this week, iPhone, Firefox, Safari, and other mobile operating systems and browsers were proven vulnerable to zero day exploitation. (The Register Article). Many people have expressed to me that txsBBSpy doesn’t actually have an infection vector and that mobile devices are secure from attack. I think the results of Pwn2Own clearly demonstrate otherwise. Mobile devices are just as insecure, if not more so than the standard desktop system. What makes it even more dangerous is that researchers who sell their exploits can get between 10K$ and 115K$ depending on the specifics of the flaw. That’s no longer chump change! Why would any researcher have any incentive at all to disclose the flaw responsibly given the big dollars that can be made by selling to a broker.

The only thing really limiting researchers from selling their flaws on the open market is the threat of incarceration. Jeremy Jethro was sentenced this week to three years probation and 10K$ in fines for selling exploit code to hacker Albert Gonzalez who in turn used the code in hacking companies and stealing 90 million credit card and debit card numbers. Gonzalez paid Jethro 60$K for the exploit while Jethro had no indication that Gonzalez intended to use the exploit code in any illegitimate way. Had this gone to court, the precedent that could have been set here is astonishing. Luckily this case was a plea bargain, otherwise the selling of exploit code would essentially be criminalized and we wouldn’t be sure to what degree this really impacts the researcher. If a researcher were to sell his exploit code to ZDI and then ZDI somehow accidentally leaks the code that is then used in an attack, who is to blame and who pays the fines/jail time? If a researcher sells his code to an independent broker who then resells the code to a criminal, who is left holding the legal bag? We do know this much.. it’s dangerous and potentially illegal to sell exploit code that is then used in a crime regardless of your knowledge of the crime. Everything else is still shades of grey.

What does this mean for mobile based Spyware? It means that those vulnerable phone operating systems and browsers are likely to get exploited with previously unknown vulnerabilities and spyware like mine is likely to be the resulting payload. Welcome to the era of malicious mobile attacks.

Veracode Security Solutions

Veracode Security Threat Guides

This widespread attack on US high tech companies signals that 2010 is the year organizations will wake up that there are sophisticated attackers after their intellectual property such as source code and hardware designs. All the same attacks used to steal CC#’s and online passwords for financial theft are being targeted at intellectual property.

Attackers are well organized and have command & control in place so that the discovery of a zero day vulnerability can be used to maximum advantage by rapidly hitting a large number of high value targets.

The only solution to running software with latent vulnerabilities is to stop running software with latent vulnerabilities. Anti-virus and IDS won’t help when it is a zero day vulnerability where there is no pattern to match. Software acceptance needs to include evidence that rigorous security testing was performed.

It is time for organizations to take a hard look at the set of client software they allow on their employees workstations and determine how trustworthy that software is. In most organizations these client systems have unbounded risk and are receiving data from the untrusted internet. If this doesn’t change, attacks similar to what happened to Google are going to effect every organization with something of value.

Veracode Security Solutions

Veracode Security Threat Guides

I checked out the lxlabs product documentation and website and could not find any reference to using a secure development lifecycle. I did find this rather disturbing post to their forum as the first post on a new topic “Security” in April 10, 2009. It was not replied to until yesterday.

http://forum.lxlabs.com/index.php?t=msg&th=11197&start=0&:

Lxadmin/hyperVM has become popular enough that people are SPECIFICALLY targeting these softwares now, and so we are now redoubling our focus on security.

If anybody knows about any vulnerability in hyperVM or lxadmin, please contact lxinfo at lxlabs.com, and we can negotiate a payment if you can demonstrate it clearly.

Of course, after we fix the bug and update the softwares, we will absolutely disclose it publicly too, since we believe in 100% openness, but we need to know about vulnerabilities before it can impact our clients.

Thanks.

This is obviously not a good software security strategy. The owner of the IP is responsible for testing for security flaws. It was obviously too little too late for lxlabs. The industry can learn from this lesson. Don’t wait until your software reaches critical mass and raises the attention of blackhat researchers before you start to think about application security.

The bigger issue and one that Vaserve should be asking itself is why did they place so much trust in software that clearly didn’t have a software security process behind it. Vaserve should have looked for evidence of a 3rd party security review before they accepted the risk of an application that has the potential to bring down their whole company.

Hosting and cloud provider customers need to ask themselves how they vet the providers they use. Have their providers demanded evidence of 3rd party security reviews of the products in their infrastructure stack? Until customers start requiring this evidence these disasters will continue. Evidence of a security review has to start with the end user customer and work its way up the supply chain to hosting/cloud provider and then to the software vendor.

Veracode Security Guides

Data Security Resources

We recently learned our database was illegally accessed and certain contact and account data were taken, including Monster user IDs and passwords, email addresses, names, phone numbers, and some basic demographic data. The information accessed does not include resumes.

Considering the well-known tendency to use the same password on multiple websites, compounded with the fact that Monster pledged a comprehensive security review after the first breach, it’s just embarrassing that they are still storing passwords in the clear.

So let’s talk about how to properly store passwords for a web application.

Use a one-way cryptographic hash

Don’t store your passwords in the clear! If you do, an attacker just needs to find one SQL Injection vulnerability and he’s got the password for every one of your users. The idea behind using a one-way algorithm is that the hash value can’t be reversed to “decrypt” the password. So how does authentication work? When a user attempts to login, you apply the same one-way algorithm to convert the user-provided password into the hash value, and then compare the two hashes. If they match, then the user-provided password was correct. At no time is the password ever stored in the clear.

Often, developers will hear the advice “use a hash” and interpret that as “run the plaintext password through MD5 or SHA-1 and store the result.” But that only solves part of the problem — the part about using an irreversible algorithm. It doesn’t protect against pre-computation. Let’s say you’ve used SHA-1 to hash your passwords, and your USERS table looks like this in the database:

USER          PASSWORD_HASH
admin         5baa61e4c9b93f3f0682250b6cf8331b7ee68fd8
bob           fbb73ec5afd91d5b503ca11756e33d21a9045d9d
jim           7c6a61c68ef8b9b6b061b28c348bc1ed7921cb53

So if you wanted to obtain the original passwords you’d have to run a dictionary or brute force attack, hashing all possible password options with SHA-1 and comparing the output to the stored hashes. This would take a long time but eventually you’d figure some of them out. But what if you already had a list of all 8-character permutations and their corresponding SHA-1 hashes? Now all you have to do is look up the hashes, rather than computing them on-the-fly. This is the idea behind rainbow tables.

An attacker with a SHA-1 rainbow table covering 8-character alphanumeric combinations would quickly look up those three hashes and obtain the original passwords of “password”, “p4ssword”, and “passw0rd” respectively.

Use a salt

The best defense against pre-computation of raw hashes is salting. To salt a password, you append or prepend a random string of bits to the plaintext password and hash the result. You then store the salt value alongside the hash so that it can be used by the authentication routine. Look in the /etc/shadow file of any modern Unix system and you’ll see something like this:

user1:$1$lKorlp4C$RD5TSM6PaZ6oaWRVUuXT40:13740:0:99999:7:::
user2:$1$qOmA0CUm$I6IdbZDTDl6B6m7s77VPe1:13650:0:99999:7:::
user3:$1$nIEInNo5$PSxcLtvGIJArL8r2AQl74.:13749:0:99999:7:::

Let’s look at the “user1″ entry in the example above, paying attention to the second field which contains a bunch of alphanumeric characters separated by dollar signs. The first token, 1, is a version number, The second token, lKorlp4C, is the salt. The third token, RD5TSM6PaZ6oaWRVUuXT40, is the one-way hash that was calculated using lKorlp4C as the salt.

When the user attempts to login, the system passes the user-provided password along with the stored salt into the hash routine (in this case, md5crypt), and compares the result to the stored hash.

Each bit of salt used doubles the amount of storage and computation required for a pre-computed table. For instance, if we used one bit of salt — either 0 or 1 — the rainbow table would have to account for two variations of every password. Eight bits of salt require 2^8, or 256 variations of every password. Use a sufficiently large salt and pre-computation becomes infeasible. For example, the md5crypt utility uses 48 bits of salt (and for an extra layer of protection, it runs 1000 iterations of MD5 to slow down dictionary attacks).

There are a couple of common mistakes that people make with regard to salting. First, don’t use the same salt every time. If you do, you’re not really increasing the search space because the attacker only has to account for a single salt value. Second, don’t worry about protecting the salt values, they’re not secrets. The added security is derived not from the secrecy of the salt but rather by the amount it increases the resources required for pre-computation.

If you have OpenSSL installed you can play around with various salt mechanisms and see what the output looks like:

$ openssl passwd -h
Usage: passwd [options] [passwords]
where options are
-crypt             standard Unix password algorithm (default)
-1                 MD5-based password algorithm
-apr1              MD5-based password algorithm, Apache variant
-salt string       use provided salt
-in file           read passwords from file
-stdin             read passwords from stdin
-noverify          never verify when reading password from terminal
-quiet             no warnings
-table             format output as table
-reverse           switch table columns

$ openssl passwd -1 password
$1$LH1SwzJI$0ho4XuPVfGlbWIcNuGIap/
$ openssl passwd -1 password
$1$eAUtQOBh$GlvJwVsyb8In5KKkvnR0E0
$ openssl passwd -1 password
$1$PgaSiWTy$ElLh6uy83Y6T4Y70AGmV20

A quick Google search shows that there is a lot of confusion about salting.

But wait, now my password recovery feature won’t work

What’s that? You say your application has one of those “Forgot My Password” features where a user can type in their username and their current password will be sent to the e-mail address on file? Clearly, that requirement depends on passwords being stored either in the clear or using a reversible mechanism such as symmetric encryption.

The answer here is to redesign your password recovery feature. Don’t let an unnecessary requirement force you into poor security practices. If you must e-mail a password, generate a temporary password that’s only valid for a short time period, and require the user to login immediately and select a new password. This obviates the need to retrieve the original, forgotten password.

Why not just use symmetric encryption?

Instead of storing passwords in the clear, you could encrypt them using a symmetric algorithm such as AES and have the application encrypt/decrypt as needed. While this solves the plaintext storage problem, it creates a new problem: key management. Where do you store the key? How often does it change? How many people have access to it? What do you do if/when the key is compromised? And so on. The tradeoff really isn’t worth it for something that’s more elegantly solved with salted hashes.

Layered defenses

While you’re rethinking password storage, it might be a good time to consider other common flubs such as password complexity and brute-force protections.

In conclusion

• Storing passwords in the clear puts your users at unnecessary risk if (when) your application database is compromised
• Use salted hashes instead of storing passwords in a recoverable format
• Password recovery mechanisms can be implemented without needing to obtain the original password
• As with any aspect of security architecture, use layered defenses

Have fun refactoring!

Veracode Security Solutions

Veracode Security Threat Guides

In 2000, I wrote an advisory about NTLM relaying (CVE-2000-0834). The problem turned out to be significantly larger than I originally suggested in the advisory. The attack extended to other NTLM-based authentications on other protocols and allowed general-purpose credential theft via a man-in-the-middle attack.

The SMBRelay tool was published in 2001 by Sir Dystic of Cult Of The Dead Cow, and that really took it to the next level. The protocol completely fell apart. It kicked off a number of other analyses of the NTLM protocol that finally resulted in this patch. Eight years after it’s discovery.

At least they got around to it. Thanks!

Veracode Security Solutions

Security Threat Guides

Almost as bad as the Sprint PCS password reset fiasco that made the news in April, here is the Yahoo Mail password reset screen:

As you can see, you need to know the user’s birthday, country of residence, and postal code. Not difficult information to dig up in Palin’s case, as shown here. After you enter this information correctly, you are asked to type in the alternate e-mail address that’s associated with the account. But they give you hints — so if your alternate e-mail was sarah@alaska.gov, they would show you s****@a*****.gov.

Assuming you guess the alternate e-mail correctly, Yahoo mails a password reset link to that address. So it’s likely that the attacker may have also had to gain access to her alternate e-mail account. Either that, or they exploited a vulnerability in the Yahoo password reset mechanism itself, which seems less likely but not implausible.

So Yahoo itself probably didn’t get hacked, per se, even though there will probably be a lot of FUD in the media about that.

Update 08/18/2008 1:00am EST:

Just found this writeup describing how it transpired: http://pastebin.com/f7fb944c5. Again, not vouching for the authenticity but it does seem plausible, and it’s consistent with my password reset theory. I guess my Yahoo account doesn’t have a secret question defined so I wasn’t presented that option when I tested the reset mechanism earlier today.

Just for fun, here’s the list of non-customizable secret questions Yahoo lets you pick from, as of tonight:

And they sure don’t make it easy for you to update your secret question, do they? (must be logged in to Yahoo for that link to work)

Veracode Security Solutions

Veracode Security Threat Guides