What’s interesting to me about this breach is that Zappos is renowned for their customer service, so watching how they communicate in the coming days and weeks should be an interesting case study. A few notable points so far:

• Tony Hsieh, the CEO, posted a copy of the internal email that they had sent to all their employees. And tweeted about it. The only difference from the customer-facing email was that it stated the number of records affected. But it’s unusual for a company to share internal communication like this.
• Zappos expired everybody’s password, forcing customers to follow the password reset workflow before regaining access to their account. Usually a company will urge you to change your password but won’t force you to do so. This was a good move on their part. The servers seemed very overloaded though; around 9pm last night it took me a few minutes (and a couple of server timeouts) to successfully reset my password.
• Around the same time, Zappos disabled international access to their website, meaning that anybody outside the US couldn’t reset their password as instructed in the email. This seemed a bit odd. As I am writing this post, the site is still unavailable to international customers. This has understandably frustrated some customers.
• In the customer-facing email, Zappos notes that credit card numbers were not affected, but “cryptographically scrambled” passwords were. This is where I believe they ought to be more forthcoming. What does “cryptographically scrambled” entail? An unsalted MD5 hash, Stratfor style? Salted hashes? Symmetric encryption? A homegrown algorithm? Something stronger like bcrypt or scrypt? This detail is critical, because it indicates how easy it will be for attackers to recover the original passwords from the affected customers and try to use them on other sites like Gmail, Facebook, Twitter, and others. Customers might be more likely to change their password on other websites if they understood the relative risk.
• The email does not disclose how long customer data was exposed prior to the breach notification. This is an important detail that was omitted.
• Zappos has been actively engaging with customers on their @Zappos_Service Twitter account. In fact, last night when I posed a question to the CEO’s Twitter alias, @Zappos_Service responded 4 minutes later. They didn’t have an answer, but they responded.
• They turned off their phone system because they felt responding via email would be more efficient (and their phone system couldn’t handle the volume anyway). Still, can you imagine a “typical” company doing this? It seems simultaneously crazy and brilliant.
• It takes a long time to send 24+ million emails. I received mine last night at 8:34pm and 9:03pm, but a colleague here at Veracode mentioned he didn’t get his until this morning. So assuming they’re going out alphabetically by e-mail address, that’s how long it took to get from “c” to “r”.
• Since both Zappos.com and 6pm.com were affected, it’s possible that they shared a single database. There are a bunch of scenarios though. It could be a vulnerability in application code shared by both sites. It could have also been an insider attack, but the fact that credit card numbers were not compromised suggests to me that the attack was external.

For me, the two things to watch for now are how quickly they restore international access and whether or not they disclose how passwords were stored and what “cryptographically scrambled” means in that context. Security breaches happen to the best of companies and these days what differentiates you is how you respond. So far I believe Zappos is on the right track.

(Incidentally, Tony Hsieh’s book, Delivering Happiness, is a fantastic read. I have a lot of respect for how this company operates, and not just because my shoes arrive overnight.)

Veracode Security Guides

Data Security Resources

Just before the holidays, we detected a cross-site scripting (XSS) vulnerability while running a web application scan for one of our customers. Nothing special about that; we detect thousands of these things every week. But as we discussed this particular finding, we noticed that the layout of the website looked… familiar. As it turned out, the discussion forum where we found the XSS was a SaaS-based product called Lithium.

From Lithium’s website: “The world’s most innovative companies such as AT&T, Barnes & Noble, Best Buy, Sephora, Univision, Home Depot, and HP use Lithium to engage their customers in breathtaking new ways (literally, breathtaking).” Run this Google search and you’ll find a bunch of Fortune 500 companies using their software.

So now, instead of one XSS, we have hundreds. It’s not just our customer who is impacted. Suddenly it’s kind of a big deal.

Here’s how everything played out. We’ll do this timeline CORE Security style (all times EST).

• 2011-12-15 5:29 PM. We fire off a quick email to the address listed on Lithium’s Security page.
• 2011-12-15 6:12 PM. We receive a response from Misha Logvinov, Lithium’s CIO.
• 2011-12-15 6:23 PM. We encrypt and send over the vulnerability details to Lithium.
• 2011-12-15 11:40 PM. Lithium reports they have a patch ready to go and will update in the morning.
• 2011-12-16 2:30 PM. We do a little poking around and it seems the vulnerability is patched for some domains but not others. We email for a status check.
• 2011-12-16 2:37 PM. Lithium confirms that the patch is in the process of being rolled out and will be completed by close of business.
• 2011-12-16 COB-ish. We’re not seeing any more vulnerable instances.

Anybody who has reported vulnerabilities before can appreciate how unusual it is for a vendor to respond this quickly. Everything was accomplished in under 24 hours! That is practically unheard of.

From a “big picture” perspective, this whole situation illustrates some important application security themes:

• It’s a canonical example of software supply chain risk. A single XSS vulnerability simultaneously affected hundreds, maybe thousands of customers. No matter how securely these companies developed software internally, they were still exposed to vulnerabilities in third-party software.
• It emphasizes the ecosystem effect of vendor security assessments. One Lithium customer did an analysis of third-party code they were operating. A defect was found, and the vendor fixed it. Now all Lithium customers benefit, without having to lift a finger! Imagine if all companies assessed their third-party code and insisted on fixes from their suppliers.
• It shows that SaaS can have huge security benefits. Imagine if Lithium had been deployed as an on-premise product at each customer sites, requiring each customer to download and install a fix themselves. Some companies would probably never get around to patching their servers. The flip side is if the SaaS company dragged their feet — or simply refused — to patch the software, leaving the customer without a viable mitigation.
• It demonstrates that application security response can be done right. Lithium engaged quickly, took the vulnerability report seriously, and wasted no time in fixing the problem. It’s not uncommon for some vendors to take months.

Increasingly, we’re seeing our customers rethink how they vet the software they purchase or license from third-party suppliers. I hope success stories like these become commonplace as we start holding software suppliers — both SaaS and on-premise — accountable for security, not just functionality.

Veracode Security Guides

Data Security Resources

At CanSecWest security conference this week, iPhone, Firefox, Safari, and other mobile operating systems and browsers were proven vulnerable to zero day exploitation. (The Register Article). Many people have expressed to me that txsBBSpy doesn’t actually have an infection vector and that mobile devices are secure from attack. I think the results of Pwn2Own clearly demonstrate otherwise. Mobile devices are just as insecure, if not more so than the standard desktop system. What makes it even more dangerous is that researchers who sell their exploits can get between 10K$ and 115K$ depending on the specifics of the flaw. That’s no longer chump change! Why would any researcher have any incentive at all to disclose the flaw responsibly given the big dollars that can be made by selling to a broker.

The only thing really limiting researchers from selling their flaws on the open market is the threat of incarceration. Jeremy Jethro was sentenced this week to three years probation and 10K$ in fines for selling exploit code to hacker Albert Gonzalez who in turn used the code in hacking companies and stealing 90 million credit card and debit card numbers. Gonzalez paid Jethro 60$K for the exploit while Jethro had no indication that Gonzalez intended to use the exploit code in any illegitimate way. Had this gone to court, the precedent that could have been set here is astonishing. Luckily this case was a plea bargain, otherwise the selling of exploit code would essentially be criminalized and we wouldn’t be sure to what degree this really impacts the researcher. If a researcher were to sell his exploit code to ZDI and then ZDI somehow accidentally leaks the code that is then used in an attack, who is to blame and who pays the fines/jail time? If a researcher sells his code to an independent broker who then resells the code to a criminal, who is left holding the legal bag? We do know this much.. it’s dangerous and potentially illegal to sell exploit code that is then used in a crime regardless of your knowledge of the crime. Everything else is still shades of grey.

What does this mean for mobile based Spyware? It means that those vulnerable phone operating systems and browsers are likely to get exploited with previously unknown vulnerabilities and spyware like mine is likely to be the resulting payload. Welcome to the era of malicious mobile attacks.

Veracode Security Solutions

Veracode Security Threat Guides

This widespread attack on US high tech companies signals that 2010 is the year organizations will wake up that there are sophisticated attackers after their intellectual property such as source code and hardware designs. All the same attacks used to steal CC#’s and online passwords for financial theft are being targeted at intellectual property.

Attackers are well organized and have command & control in place so that the discovery of a zero day vulnerability can be used to maximum advantage by rapidly hitting a large number of high value targets.

The only solution to running software with latent vulnerabilities is to stop running software with latent vulnerabilities. Anti-virus and IDS won’t help when it is a zero day vulnerability where there is no pattern to match. Software acceptance needs to include evidence that rigorous security testing was performed.

It is time for organizations to take a hard look at the set of client software they allow on their employees workstations and determine how trustworthy that software is. In most organizations these client systems have unbounded risk and are receiving data from the untrusted internet. If this doesn’t change, attacks similar to what happened to Google are going to effect every organization with something of value.

Veracode Security Solutions

Veracode Security Threat Guides

• Stack buffer overflow in URL blacklisting code due to a fixed-length buffer, triggered by URLs longer than approximately 2064 characters
• Stack buffer overflow in filter file parsing due to a fixed-length buffer used in a call to fscanf()

In addition, the Michigan team noted that the software fails to encrypt or authenticate the filter auto-update process, and the use of unsafe string processing functions is systemic, meaning that other exploitable vulnerabilities may be lurking just beneath the surface.

Upon learning of these vulnerabilities, the Chinese government ordered Green Dam to fix the security holes immediately. But even with those hastily applied patches, it seems likely that the software is probably riddled with additional flaws. In downplaying the severity of the entire matter, Green Dam implies that their development process probably doesn’t include independent, third-party security assessments. If quick fixes of the most severe issues are sufficient to appease the government, that is probably all they will do.

Ironically, by attempting to “protect” Chinese citizens from online content, the government is doing exactly the opposite by reducing the security posture of those PCs and homogenizing the attack surface. You can just envision all the foreign governments and botnet operators rubbing their hands together with glee as they prepare to fuzz Green Dam for some 0-day exploits. The government wants to be perceived as caring about Internet safety (hence the public insistence on the bug fixes) but in reality they are adding a weak link to the chain.

The Green Dam software is available for free download.

Veracode Security Solutions

Security Threat Guides

If you haven’t taken a crack at it yet and plan to, then stop reading — SPOILERS AHEAD, as they say. Otherwise, I hope that this is helpful to anyone who is interested in learning more about basic cryptography.

I started by copying and pasting the binary digits from the cover of the report into a text file. Then, I converted the bits to ASCII resulting in the following text:

$ cat vz|unsplit|bin -d|split 72
EVNTXIGYIMWSNEHEIEFOTXBSCWYHRQMWGUZABVYCBBFREYFBVEDKEVMFRIFNGFNRBFGVKSFP
NBUFZJGCEEEWAKHPXEBTZJCZOWGTBSQGTMIAYDPYDRIRYETKCJRPYHEPWKUOAEKNVTVZHSMZ
NTTIVIKMMRYSNUIAKBRKQMSTYCGCCRLRRIIREFGYTJUBUXHEYSGLEYRVHIYXDEYZCJKVTOSO
IXJEHOXEVMWJBNZMTKWZEFOFCNBWNCUWMYFIUVBKWNPWTYOEYQTIRRYRCMNVFVLRSBNTPWPA
OCZPEKHLFCEERRVWVUYBVJPUVPOAYMIKQQNSWZGHZKDGYLAEGWPKESGCYZFVJDMEPQKSSLNV
SVPUVVRVYERHDTUTYYMQGEVWRMQSZFNPNRJIGGWAJNNJLKOEQHNETRPUQYDFZWCZKVJEXLMC
KCSIFTCTSUTLDRRMIKQTNINPGRPQQXPTZDPAIOTCEUAZFEWDQLLPZRHXLXQGSLRJTBLZRIRV
ISNZIWLMVYADVOHFEVNAKKGORRXSYGXPUMVGBOMRJLCREFCMRQVXTMIYMJJVHXNBTSZMTJEF
KFGKURFLNHXPKCWLEXMIYLGYNNRWAKSEWTHPKGZKKXGAZELLUTAYCIEKWISHUNDKEKWARGBY
ZFGKEPKQGZZSRIMFLGKARTURAINSNGEEUMEXRVEELZXTISUWVZKOYLTPBHZWEOQWNXNPXPKS
SXJHPANCVFPRYADRLROEWEBQEWHZRGATZDGUCEKLFYHZJNNZIJRGNZRVBOCAUYEZGKPSJXJI
ASMVFTDWFXBIDHQZEYKDRTDRIOPPKJRPISSKMCZJFZTBVBJUGEYANJIGJTDCPTZDEOGUTLZP
EKHTNIHTGGUMVGBOMRJLCREFSWFZOCROHEAU

The first thing I tried was Caesar shifts, which is basically ROT-n for values of n from 1 to 25. So for n=1, A is encoded as B, B is encoded as C, and so on, all the way through Z, which is encoded as A. For n=2, A is encoded as C, B is encoded as D… you get the picture. I won’t print out the results of all the decodes because it would take up too much space, but suffice it to say, nothing interesting came out of it.

Next, I tried frequency analysis, which can be an effective way of deciphering a simple substitution cipher (i.e. a given plaintext character is always encoded as the same ciphertext). A simple substitution cipher reflects the tendency of a written language to use certain letters more than others. For example, in English, the most frequent letter, E, appears roughly 170x more often than the least frequent letter, Z, for a sufficiently large sample size. Here’s the frequency distribution from the Verizon ciphertext:

$ cat vz|unsplit|bin -d|split 1|sort|uniq -c|sort -n
     21 D
     21 Q
     24 O
     25 X
     26 A
     26 H
     27 B
     28 L
     28 U
     29 J
     31 C
     32 M
     33 W
     34 F
     34 S
     36 P
     38 I
     40 N
     40 V
     40 Y
     41 G
     42 Z
     43 T
     44 K
     56 R
     61 E

As you can see, the most frequent character, E, was only three times as prevalent as the least frequent character, D, which meant it was unlikely to be a simple substitution cipher, provided the plaintext was English. The frequency distribution was far too different than what we would expect.

Just for kicks, I tried various transposition ciphers, rearranging the 900 characters into an M-by-N grid, for different values of M and N (M*N=900), and reading down the columns instead of across the rows. Frequency analysis already told us that we shouldn’t expect to see any English text, but I thought some visual patterns might emerge. Wrong again.

Around this point, I saw somebody on Twitter mention that there were clues embedded in the body of the report, so I started skimming through it. At the bottom of page 48 is “yr puvsser vaqrpuvssenoyr” which ROT-13 decodes to “le chiffre indechiffrable.” Here’s where I went briefly astray by using Google Translate instead of just Googling the term. The literal French translation is “indecipherable figure” which made me think that the clue was that the whole thing was a hoax and the front cover was just a bunch of garbage. A friend reminded me that “le chiffre indechiffrable” actually refers to a Vigenère cipher, which would have been painfully obvious if I’d used regular Google search instead of Google Translate (smacks self on head). Logically, the Vigenère would’ve been the next target anyway, as it’s just a simple substitution cipher with a twist.

If you’re not familiar with how a Vigenère cipher works, it basically uses a keyword to cycle through different substitution maps. For example, if you were encoding ZZZZZZ with the keyword FOOBAR it would come out as ENNAZQ — the letter Z is encoded differently depending on how it aligns with the keyword. You can see why frequency analysis isn’t useful here.

My first inclination was to just guess the keyword outright. I thought maybe it was something obvious such as VERIZON, VZ, RISK, DATA, BREACH, VIGENERE, etc. I grabbed Crypt::Vigenere and tried each of the guessed keywords, but none of them worked. I even wrote a quick script to brute force all 2- and 3-letter keywords, again coming up with nothing.

Then I took a different approach — trying to guess what the decoded message might contain and work backwards. I speculated that the first word would be CONGRATULATIONS which corresponds to a potential key of CHANGINEXMDKZRP. This didn’t seem right, but the CHANGIN part of it seemed like too much of a coincidence. So I tried CONGRATS as the plaintext, which corresponded to the keyword CHANGING. I thought it was solved at this point, but decoding the entire ciphertext using CHANGING as the keyword still gave me junk. So then I searched through the PDF for the word CHANGING, and sure enough, on page 46, one of the bullet items says “Changing default credentials is key” (clever, huh). So I decoded with a keyword of CHANGINGDEFAULTCREDENTIALS and it worked. The text decodes to the following message:

$ cat vz|unsplit|bin -d|vigenere -d changingdefaultcredentials|split 72
congratsfirsttocrackgetsrewardgotowwwverizonbusinesscomslashdbirhunttocl
aimforeveryoneelsehighlvlstatsforfinsvcsandretailfollowplssharefinsvcsso
urcesexternalnineteeninternalninepartnertwothreatsmalwareelevenhackingfi
fteendeceitfourmisusesixphysicaltwoerroroneerrorsigcontributorinfifteent
opthreehacktypessqlinjectionsevenmisconfigaclssevendefaultcredstwotophac
kvectoriswebapptentopassetisonlinedatatwentysixandallrecordstopthreedata
typesauthcredelevenpiitenpymntcardeightpymntcardwasninetyeightpctofrecor
dstopuuisunknownconnectionssevendiscoverytakesweekstomonthsretailsources
externaltwentythreeinternalonepartnereightthreatsmalwaretenhackingtwenty
onedeceittwomisusetwophysicalzeroerrorzeroerrorsigcontributorinsixteento
ptwohacktypessqlinjectionsevenstolencredsseventophackvectorisremaccmgtei
ghttopassetisposelevenandoverhalfofrecordstoptwodatatypespaycardtwentyth
reepiininediscoverytakesmostlymonths

Had the message not begun with “CONGRATS”, there are some other techniques for attacking a Vigenère cipher, including trying to deduce the length of the keyword by looking for cyclical patterns in the ciphertext. Luckily, it didn’t come to that because I wanted to watch TV.

I visited the embedded URL which said that somebody had already claimed first prize but that I was still in the top three. I later found out that about ten people, including myself, submitted solutions around the same time before the authors could update the congratulatory message. So I didn’t win any money but it was still a lot of fun (and significantly better that the corny FBI challenge).

Veracode Security Solutions

Veracode Security Threat Guides

In 2000, I wrote an advisory about NTLM relaying (CVE-2000-0834). The problem turned out to be significantly larger than I originally suggested in the advisory. The attack extended to other NTLM-based authentications on other protocols and allowed general-purpose credential theft via a man-in-the-middle attack.

The SMBRelay tool was published in 2001 by Sir Dystic of Cult Of The Dead Cow, and that really took it to the next level. The protocol completely fell apart. It kicked off a number of other analyses of the NTLM protocol that finally resulted in this patch. Eight years after it’s discovery.

At least they got around to it. Thanks!

Veracode Security Solutions

Security Threat Guides

Partial Disclosure – The Good

The responsible disclosure process tends to break down in rare occasions where the vendor doesn’t want to fix the issue. When this occurs, the researcher is put into a difficult position whereby full disclosure could put users’ systems at high risk of compromise. The other case where partial disclosure becomes an alternative is when the researcher has discovered a design flaw in a protocol or underlying multiple vendor component. Examples of this case include the DNS flaws published this past summer by Dan Kaminsky and the TCP denial of service condition discovered by Robert E. Lee and Jack Louis that is currently in the disclosure process. When the flaw affects a very large number of vendors and the actual problem is located within the underlying protocols that support the communications of the Internet as a whole, one possible solution is to follow a partial disclosure model where phasing the details to the general public can be used to encourage adoption and creation of patches throughout the enormous target audience.

Partial Disclosure – The Bad

What is driving the fear surrounding partial disclosure is the potential for abuse. When a major flaw is partially disclosed, a number of potential issues may occur. First and foremost, the further along the partial disclosure path we are, the more details will be released to the public, and the higher the probability that someone (either good or bad intentioned) will figure out the exploit and disclose the details. Second, when partially disclosing, the vendor’s hand is being forced into a situation that could speed up fixes, reduce testing, and cause ripple problems elsewhere within the infrastructure. It is difficult enough to dance the fine time line when doing responsible disclosure, but if we are escalated to the point of partial disclosure, additional fuel is added to the fire.

The Ugly

The real ugly part of partial disclosure is when we add to the equation the ability to spread fear, uncertainty, and doubt into the normal user community. It is generally well accepted that FUD can be used to drive additional revenue. If it is possible to increase the perceived magnitude of the “problem” that your product or service solves, it is possible to directly impact the demand for that product or service. That is the major fear imposed by the growing trend of partial disclosure. By releasing just enough information to trigger wide scale speculation into the flaw, it is possible to create buzz and garner media attention resulting in a lot of speculation and very little hard facts around the issue. The potential for abuse by the security industry at large is enormous.

The Fix

Some have suggested a group of security researchers be convened to vet the requirement of partial disclosure and to allow for independent peer review of any security research that requires the partial disclosure process. This suggestion leaves questions regarding who would stand on this group and who would be impartial enough to ensure that the right thing was always done regardless of profit potential. It also leaves open the opportunity for member researchers to utilize the information gathered during the vetting process to position themselves to profit from the data upon release. It might be wiser to rely on a higher level authority or government entity to manage this process and use the services of security researchers as required for subject matter expertise. While a group of this type wouldn’t ensure that all partial disclosure is appropriate, it would hopefully limit the potential for abuse and the ever present chance that people try to profit from the FUD that surrounds the current partial disclosure process.

FREE Security Tutorials from Veracode

Cyber Security Risks
Mobile Security
CRLF Injection
Flash Security
SQL Injection Hack
 

Veracode Security Solutions

Software Security Testing
Binary Analysis
Application Analysis
 

Veracode Data Security Resources

Data Security Issues
Data Breaches
Data Loss Protection

What’s more interesting to me, now that I’ve digested the big secret, is how this whole situation has played out in the security community.

The security community has been polarized for the past two weeks, not so much over the technical details being withheld, but about Dan’s plea that people not speculate about the vulnerability. As many pointed out, the “bad guys” won’t stop trying to figure it out just because the “good guys” keep quiet. To be honest, my own lack of public speculation wasn’t because I agreed with the philosophy; I just wasn’t smart enough to figure out the vulnerability myself.

People implied — or stated outright — that Dan just didn’t want anyone stealing his thunder. Considering the timing of the release and the subsequent BlackHat talk, it’s obvious why such accusations were made. Personally, I think it’s a little of each. I believe the coordinated patch effort was undertaken with the best of intentions, but I also think Dan relished some of the glory and media attention as well. It’s hard to blame him for that; if you were in his shoes, wouldn’t you want some recognition too?

By many accounts, dealing with the DNS vulnerability from the operational side has been an exercise in frustration. Plenty of IT people wanted to patch but couldn’t get approval without being able to justify the operational risk. “Because Dan said so” is apparently not a convincing enough argument. Some wondered why the people who were responsible for creating the problem should be blindly trusted to implement an appropriate fix?

Ultimately, vulnerability disclosure is a minefield. No matter how you choose to disclose, somebody will always disagree.

P.S. If you didn’t figure out the title of the post by now, Nate was one of the unlucky few to draw the same timeslot at BlackHat as Dan Kaminsky.

FREE Security Tutorials from Veracode

Cyber Security Risks
Mobile Security
CRLF Injection
Flash Security
SQL Injection Hack
 

Veracode Security Solutions

Software Security Testing
Binary Analysis
Application Analysis
 

Veracode Data Security Resources

Data Security Issues
Data Breaches
Data Loss Protection

Faced with nearly a month of criticism and questioning, and understanding the persuasive power of a technical peer review, Dan decided to expand the inner circle, so to speak. Rich Mogull arranged a phone call with Tom Ptacek and Dino Dai Zovi so that Dan could spill the beans and let them decide for themselves whether it was spin or substance. Turns out there was substance.

Now we sit around and wait until August 6th to cram into a ballroom with a thousand sweaty conference-goers to hear the juicy details. And Dan’s presentations are usually packed to the brim even when he’s not announcing anything.

In the meantime… how about patching those servers?

Veracode Security Solutions

Veracode Security Threat Guides