Introducing Mr Fails

I recently came across a very entertaining Software Testing Club eBook that will make the tester in you laugh out loud! This is the sad story of Mr Fails.

Mr Fails had a problem in that whenever he touched any electronic equipment it would fail or break. As you can imagine, this made Mr Fails feel useless. He couldn’t do very much without anything breaking. For example, when ever he tried to buy a can of pop (soda) from a vending machine, the machine would swallow his money and report an error for no apparent reason. Read the full eBook here.

Mr Fails, Veracode, and the Software Testing Club would love to get your feedback.

Now, let’s examine another market. According to the 451 group, the market size for automated application security technologies (as defined by static analysis, dynamic analysis and Web Application Firewalls) is forecast to be just a little over $1B dollars in 2014 .

Is someone else feeling that these two market forecasts taken together just don’t add up?

We spend 0.3% of what we pay for software on ensuring that it is secure! Now you can argue that the application security market as defined above is a narrow representation. For example, manual testing is not included. However, even when you account for those variances the gap in what we spend to buy software and what we spend to secure it is huge.

This brings me to the market for testing third-party software suppliers that was explored in Volume 4 of the State of Software Security report. As the reliance on third-party software and components has grown, so has the awareness that security weaknesses embedded in those applications become a liability for the enterprise. This recognition transcends the security community as you see calls for this level of due diligence from leaders in the sourcing and vendor management area as well (See Forrester report, “Why Stronger Vendor Management is Essential to Managed Services Relationships” by analyst Jan Erik Aase. ). We examined which industry segments are heeding this call to action and engaging in this process with their third-party software suppliers. We found enterprises representing at least eight different industry segments.

While Software and Finance account for the majority of the dataset, companies across the spectrum are starting to hold their software suppliers accountable.

According to our customers approximately one-third of applications in their environment are characterized as third-party and two-thirds as internally developed. What we see is that 30 to 70% of code components even in so called internally developed applications are in fact third-party components and libraries. With such heavy reliance on code coming from outside an organization, a formal third-party risk assessment program becomes crucial to managing overall application risk. We recommend that all enterprises institute a policy that requires third-party vendors to demonstrate proof of independent security verification or to submit to that due diligence. We also recommend that sourcing and vendor management professionals include specific language in contracts to that end.

If we are going to spend hundreds of billions of dollars on software lets at least spend a few billion more than we do right now on ensuring that it is secure!

Excerpts in italics from Hackers Intercept FBI Call With U.K.

The Federal Bureau of Investigation said cybercriminals hacked into a cybercrime conference call between its agents and law enforcement officials overseas.

The 16-minute call was posted on the Internet on Friday. The hacker collective Anonymous claimed responsibility, though the FBI didn’t name the group and said a criminal investigation was under way.

As a security person I am not content to know what happened. I need to know how it happened. Without understanding the how, we can’t prevent it in the future. In reading the news stories it has become clear how this happened.

The FBI said the breach wasn’t made on the agency’s secure email or other computer systems. Instead it appeared to be result of a law enforcement officer overseas who was invited to be on the FBI call and who forwarded the information to his private email account, which was compromised by hackers.

Anonymous had been working to compromise the personal email accounts (gmail, yahoo, hotmail, etc) of federal agents from multiple countries. Personal accounts are MUCH easier to compromise than corporate/internal mail accounts:

• The authentication and password reset forms can be reached by any attacker over the internet
• There is typically no password strength enforcement
• Users reuse passwords and the password associated with this email account may have been compromised in another breach
• There are automated password reset mechanisms.

Anonymous successfully compromised at least one agent’s personal email account. When you have a large group as a target all you need is one weak account.

An international law enforcement conference call was scheduled to discuss the Anonymous investigation. A few dozen agents from 5 countries were sent meeting invitations over secure email channels to their internal official accounts. These invitations contained the dial in number and passcode to a conference bridge.

At least one of the agents forwarded the invitation to their personal email account. At least one of the agent’s personal email account had already been compromised by Anonymous. Now Anonymous had the conference bridge information. They dialed into the conference call. The agents running the call did not audit individuals joining the call. Anonymous was able to eavesdrop on the call and deal an embarrassing setback to the investigation.

There are a few lessons we can learn from this besides not forwarding confidential mail to personal email accounts. You need a strong password on personal email, and ideally use 2 factor authentication (like Google supports) if available. Make sure you are using the strongest password reset mechanism if there are multiple offered. Don’t use a secret question where the answer is public information or easily guesable. Paris Hilton used “What is the name of your dog?” on her T-Mobile account. Not a good choice. Finally, if sensitive information is discussed on a conference bridge, audit the people joining the call. There is a reason the service beeps when people join.

As you can see the attackers are crafty and unrelenting. You need to stick to secure operating procedures or you will be easily compromised.

1. Android users potentially hit by malware attacks: Two possible Android attacks, one, according to Symantec, due to thirteen applications from three different developers that have been collecting data and performing tasks without the user’s knowledge (Millions Of Android Users Potentially Hit By New Malware Attack, by Oliver Haslam). Another is a bug unique to HTC smartphones that allows some applications to send the user’s Wi-FI network username, password, and SSID information to a remote server for collection (HTC Android phones allow apps to harvest users’ Wi-Fi password by Zeljka Zorz).
As a footnote to this news – Google announced a new service on February 2nd, 2012 called “Bouncer” that would automatically scan Android apps for malware. Check out this post by Elinor Mills at CNET to learn more.

2. Government web applications contain the most vulnerabilities by the SC Magazine Staff (@scmagazineUK). After carrying out over 600 penetration tests on custom-built applications, Context Information Security found that UK government web applications contained the highest number of vulnerabilities. Interestingly here at Veracode we have also seen similar patterns in the US and we blogged about this earlier this year.

You can download the full State of Software Security, Volume 4 report here.

3. Twitter Censoring Tweets in Various Countries: Twitter Censorship Movie Sparks Backlash: Is It Justified? by David Kravets (@dmkravets). By announcing Thursday that it would exercise its ability to withhold content from users in a specific country, Twitter sparked a massive debate in which participants toyed with the ideas of a company abiding by the law, the responsibilities of the messenger, and freedom of speech.

4. The DMARC coalition bands together to stop phishing: Google, Facebook, and Others Join to Write New Email-Authentication Spec Called DMARC by Dennis Fisher (@DennisF). Google, Yahoo, AOL, Microsoft, and others have joined forces in order to develop a new framework for email. The new specification will be called the Domain-based Message Authentication, Reporting, and Compliance, and aims to stop phishing schemes and other email-borne attacks.

5. Committee in the UK pushing for cyber security education, awareness campaigns, secure public sites, and safety standards on software: Demand for safety kitemark on software stepped up by John Leyden (@regvulture). Political types on the Science and Technology Select Committee in the UK have called for the expansion of the Get Safe Online and similar campaigns, in order to dispel fears and encourage secure usage behaviors on the Internet. Perhaps the most significant of the demands is for, “safety standards on software sold within the EU, similar to those imposed on vehicle manufacturers.” Also be sure to check out the comments.

6. Finally, this weekend is home to Superbowl! The New England Patriots will be taking on the New York Giants this Sunday in Indianapolis, but what may not be as apparent as the fans, food, and commercials is the security. By utilizing defense contractor SAIC, an $18 million Regional Operation Center, a Mobile Command Center, and even gamma-ray scanners, this Superbowl will be the most technologically secure in history. Game On: Gamma Ray Scanners To Guard ‘Most Technologically Secure’ Super Bowl Ever by Christopher Brook (@threatpost)

Wendy Nather from the 451 Group and Veracode’s CTO Chris Wysopal presented the latest research on enterprise security spend, and discussed how to “make the case” for security initiatives in a recent webinar. This popular webinar also generated a large number of questions from attendees, and the highlights of the Q&A session are posted below. You can access a full recording of the webinar here.

For those of you who missed the webinar but still have questions or comments, we’d love to keep the conversation going, please leave your remarks!

Q: How would you recommend that security professionals engage the development community about security testing?

Wendy Nather: I’ve always been a fan of bribery myself, “constructive bribery”, pretty much anything that works. Make no mistake; what you’re talking about here is really a form of social engineering… it really helps if you sit down with the developers and show them that you have the same goals as they do, and show them that you can possibly be of help to them in achieving their goals. If you do this they’re going to be a lot more receptive to any changes you’re going to ask them to make. Doing anything casually rather than bringing it down as an edict, starting slowly, getting to know them and their issues and applications, goes a long way as far as building a good foundation for working together.

Q: If WAFs (Web Application Firewalls) are as problematic as you say, why is this one of the fastest growing Application Security technologies? It seems like a WAF is a no-brainer to put up until you fix the underlying problem, isn’t it better than just being exposed?

Wendy Nather: You are absolutely right – it does seem like a no-brainer, and at least in our market place it is the fastest growing segment partially because it is so straightforward. It is a lot easier to buy technology than it is to go in and fix legacy code. The problem is not that you buy the web application firewall and you slot it into your network and try and figure out how to pipe all yourtraffic through it, the problem comes when you start changing it. It’s not binary, turning it on or off… there’s a lot of interpretation in the application and specific tuning that needs to be done and it’s there that we see a lot of enterprises dropping off the effort.

Q: What approach do I take if the majority of my applications are outsourced and I work for a global company?

Wendy Nather: That’s always been a big problem … people are realizing that software security applies across the board. One thing you can do is make good friends with your procurement team and if you don’t already have security language in your contracts with your third party providers, it’s time to try and get some. I have actually managed to get into contracts stating that the vendor would take care of any discovered security problems at their own expense, regardless of when the problem was found for the life of the contract. You’ll be surprised at how many vendors don’t read the contract before they sign it and that sort of thing! At least going forward you can start to put more weight legally to enforce these. With things you already have in place you can threaten to go to the competition because they are more secure. There is a lot of unseen power in the hands of consumers, and if they put that together the market will generate a lot more than there might have been.

Q: Per the title of the talk, how do you monetize the concepts you’ve been presenting?

Wendy Nather: How to monetize the concepts – have to go back and agree with you Chris – groups like Denim Group have actually been doing this together with other companies… the problem is that until you know the extent of what you are actually dealing with you don’t know what the expenses are going to be. You may want to start budgeting for one or two full out re-writes, and if you’re lucky they don’t have to be rewritten and you can use that budget to address some of the more common problems across the board. But knowing how much money you’re going to be spending upfront is a challenge until you have the application inventory, until you know what your risk tolerances are, and until you have a fair idea of what the problems are. You’ll have to start slow and realize the number may grow to a certain extent before you really know what you are doing.

Q: You mentioned a disparity between what is getting attacked – for example, applications – and where the money is being spent, like on networks. Why do you think that is and what can be done to correct the imbalance?

Wendy Nather: Again, network security and OS layer security have been around for a long time, people understand it well, even IT executives and business executives have a pretty good idea of what it entails. They say, “Can’t we just put a firewall in here?” That’s pretty well understood. But the problem is the implications of addressing application security are so customized per enterprise and for the types of application that they have, it’s just not as straightforward. So for the reasons that I explained before, there’s a perception that this is hard. There a lot of unknowns in it before you start and I think that’s why it hasn’t been widely adopted. But, certainly taking baby steps as Chris described and starting just to get the lay of the land and start to talk about it – because talk is cheap – and trying to raise awareness there are a lot of things you can do on a small budget to start.

You can access a full recording of the webinar here.

Consider mobile applications, there is the software that runs on your device which often connects to more software and data resident somewhere on the web. So is that one application or two?

Similarly, most web applications have three basic tiers – web tier for presenting the pages, application tier for functionality and database tier to hold the data being created and updated. Each tier consists of a combination of software the enterprise developers create, commercial off-the-shelf software and software from 3rd parties such as open source or an outsourced development company. The whole website may be counted as a single application if you are running a dynamic scan, however for static analysis you may have to test the different components as separate applications.

Enterprise applications can be even more complex! They can include integration applications that enable developers to reuse existing business applications within new applications. For example, an application for deciding whether to offer a ‘good customer’ discount on your online order could use an application to connect to an application that checks your current account balance, an application to look up your past orders, and an application to check current promotions. To understand the security posture of your ‘good customer’ discount application you really should understand all of those connections as well – but by now even I have lost track of how many applications are involved in that single transaction.

This is why Veracode’s definition of an application focuses on size rather than architecture. We define an application based on a collection of software components that deliver a business function. Our definition gives our customers flexibility in what they choose to scan. For example, customers can scan businessunit1.mycompany.com as an application. They can scan a commercial off-the-shelf package prior to purchase as an application. They can also scan a collection of software that includes an off-the-shelf package, custom developed code and open source libraries as an application. The bigger the total size, the more ‘applications’ you scan.

I should also note that for us application scanning is concerned about the executable aspects of the entire package – so we don’t count some application components towards the total size. In many cases, applications contain operating system libraries, graphics and other non-executables that do not count towards the application’s size.

When companies have a good handle on their application inventory, and what that inventory contains, it can be fairly straightforward to answer the ‘what’s an application’ question. Yet many large enterprises struggle with this because their applications are complex, their development teams change frequently, business units are consolidated, and they have new acquisitions all the time. This means they often do not have a good inventory list of applications, nor a good understanding of how applications are linked.

We’ve seen programs get stymied because the enterprise tries to answer the question by having an army of software consultants wandering the halls talking to various IT and business people and digging through the infrastructure. Since Veracode isn’t interested in hosting armies of any type, we use a combination of automated discovery techniques and work with existing organizations to design a strategy to create the list and accurately size the applications they should be testing.

For example, we worked with a global enterprise with four geographically disparate business units, each with their own application infrastructures, IT organizations and code development practices. We worked with the enterprise to organize local ‘application security experts’ aligned with each geography BU and the local CISO. Veracode trained the experts to identify and size the right applications for this program (ie, important for BU’s mission, active development was occurring, etc.). It was a lot of work just to answer the ‘what are our applications?’ question, but it was worth it. Without that effort the application security program may not have been so successful. They would have scanned applications that the security team knew about instead of applications that were business critical. In addition, that work is being leveraged in many other initiatives and strategic planning activities, because now the enterprise has both a well defined list of their most critical applications.

The same is true for complex website portfolios. Enterprises typically use our discovery services to get a complete and accurate assessment of their entire website portfolio before launching into a full-fledged web application testing effort. For example, one of our customers handed us a list of over 30,000 IP addresses and domain names as a starting point for our discovery process. We ended up testing about 3,000 web applications (with Dynamic^MP we did it in only 8 days, but that’s a story for another time). The project was deemed doubly successful since the enterprise was able to discontinue a number of defunct web properties which were still active.

For us this is further proof that answering the ‘what is an application’ question can help you get early successes with your application security program.

How does Veracode look for the presence of frameworks in Java code? Because our customers upload the application packages that they deploy or distribute (as EARs, WARs, or JARs), we can observe the presence of framework classes, configuration files, and other artifacts in the application. We record the prevalence of the framework so that we can mine the anonymized data later. We resample the data every few months to get an idea of relative framework prevalence and to see if any trends can be observed.

Below is our most current Top 10 list for Java frameworks. This list is based on a sample of over 5400 customer applications and was sampled on December 7, 2011. Note that we have decomposed one of the larger framework families, Spring, into its component frameworks to get a better idea of the usage of its individual parts. The percentages reflect the number of Java applications (not individual scans) in which the framework was observed, so an application that was scanned multiple times only counts once in the rankings.

1. Spring MVC (23%)
2. Struts 1.x (15%)
3. Apache Axis (15%)
4. Apache Xerces (14%)
5. Hibernate (12%)
6. JDOM (12%)
7. Java Applet (8.1%)
8. Apache Velocity (7.9%)
9. Apache ORO (7.0%)
10. JAX-WS (6.5%)

A couple of interesting findings here. First, the relative prevalence of Spring MVC and Struts is unsurprising, but the fact that Struts 1.x is #2 on the list and Struts 2 is not even in the Top 10 is a little surprising. (It came in 24th in the overall rankings, in fact, showing up in just 1.8% of the Java applications scanned).

Second, it’s interesting to note that there are multiple frameworks for web services in the top ten, and that Axis appears to have an edge on popularity over JAX-WS.

Third, the relatively high number of applications scanned that contained Java applets was interesting. It’s hard to imagine that 8% of all Java applications have a customer facing applet. One is tempted to speculate that in many cases these applets are administrative interfaces to framework or server code that are left in the application distribution inadvertently or unknowingly, and thus that these represent potentially forgotten attack surfaces for the application.

We’re just starting to mine the data that we’re seeing regarding frameworks. I think that this data should be interesting to development teams looking to choose frameworks that are more widely used. From a security perspective, too, this is a useful reminder that applications rely on third party frameworks, and that some of these may come with their own attack surface (e.g. applets) that shouldn’t be forgotten when planning secure deployments.

1. New Data Protection Laws: “EU to Propose New Data Breach, Privacy Regulations” by Brian Prince (@threatpost). Over the weekend the European Union announced that they would soon be proposing new laws that would require companies that are impacted by cyber attacks / data breaches to inform authorities and customers within 24 hours. The legislation will primarily be focused on protecting online consumers by giving them more online privacy and information security rights. The EU also hopes that the proposed regulations will help simplify their data protection methods. It appears that the proposed laws will probably not go into effect for another two years.

2. SQL Injection Attacks: “Avoidable Attacks Cause Most Data Breaches” by Sophie Curtis (@SCurtiss). In this article, Sophie Curtis provides insight on the widespread lack of prevention against SQL injection hacks shown by many businesses. Curtis reports that businesses with underequipped or out-of-date cyber security methods are among the easiest targets for hackers and that these attacks cost billions of dollars while impacting millions of people annually. The article also provides insight on SQL injection attacks and measures that can be taken in preventing them.

3. Kelihos Botnet: “Microsoft: ‘Kelihos’ botnet master worked for AV vendor” by Ryan Naraine (@ryanaraine). Microsoft has identified the developer behind the “Kelihos” botnet that was responsible for countless spam emails, identity theft, stock scams, and more. According to Microsoft, the software developer is Andrey Sabelnikov, a Russian man who used to work for an antivirus/firewall/security software company. Sabelnikov has been accused of creating over 3,700 subdomains from a Czech free hosting site and using the subdomains to control the Kelihos botnet.

4. Data Privacy Day: “SSCC 81 – NCSA and Data Privacy Day” by Chester Wisniewski (@ChetWisniewski). Happy Data Privacy Day! In this article and podcast, Chet Wisniewski talks about the upcoming holiday (Data Privacy Day is officially January 28th) with Michael Kaiser of the National Cyber Security Alliance. The two discuss the role of the holiday in promoting privacy and cyber security awareness globally as well as what consumers should do to protect themselves.

5. Application Security: “Cover Your App: Five Lessons from Recent Data Breaches” by Scott Vernick (@HuffPostTech). The growing problem of cyber attacks has more and more consumers thinking about the security of their personal information online. Scott Vernick offers five excellent tips on measures consumers can take to protect their data in this article from the Huffington Post.

6. Smartphone Security: “Lookout’s New App Visualizes Mobile Security Threats As They Are Detected Around The World” by Leena Rao (@LeenaRao). As it continues to become more of an issue, we are seeing many companies releasing mobile security solutions. Earlier this week Lookout released a new app for Android users that allows them to monitor cyber attacks as they take place. The app also provides information on the top security threats that are taking place, and the breakdown of malware attacks vs. spyware attacks happening in real time. Products and applications like this will hopefully increase cyber attack awareness amongst smartphone users.

Q: Are you concerned about the merge to electronic healthcare records?

RC: Yes – part of the healthcare reform package has requirements that accelerate the reliance on electronic file records in medicine. There’s some real incentives in the bill that force the industry into doing it relatively quickly. The question in my mind is who the actor is in this case that would go after health care records. Is it a criminal or is it an espionage organization? I don’t know the motivation, but I do know that these enormous insurance companies and enormous medical centers have lots and lots of vulnerabilities because they’ve never looked systematically before and done real sophisticated security analysis – that’s the last thing a major medical center has been doing in the past. So yes it is a source of concern any time a new industry runs headlong into a reliance on IT systems it hasn’t been reliant before.

Q: Is it safe to assume that most attacks come from compromised servers? If so, are there any government agencies or companies that scan for vulnerabilities that notify that company of a server issue?

RC: The simple answer to that is no. The government does not run around scanning private company servers. In fact, unless you specifically sign up with a provider to do that, no one’s going to automatically do it for you.

Q: Would you please comment on what small businesses can do to learn more about what they can do to contribute to increasing security in their respective businesses?

RC: I’m going to say something here that may be a little counter intuitive and a bit controversial. I think small businesses should think about the cloud. I know some people say, “Oh the cloud is automatically insecure,” or, “the cloud is automatically less secure.” Well it depends on what you ask the cloud provider to do. If you’re truly a small business, you don’t have the time, you don’t have the expertise, you don’t have the money to defend yourself to the level of perhaps what you would be satisfied with. But a bunch of small and medium-sized companies going to a cloud provider together can have much better security than they can have individually. If, and this is the key thing, if they ask for it, and if they compare offerings on the criteria of a service, and of security, because if you just go to a cloud provider, they’ll say, “Oh yea, we did all of the security stuff,” and that will be the end of it. You get these situations where you get the cloud provider kind of believing it’s up to you to do your own security, and you think the cloud provider is doing it, so you have to be careful, you have to be explicit, you have to ask them what additional security you can buy from them, and how you have compare the security offerings’ among the cloud providers. But I would urge a small business owner to try to do that rather than try and secure it themselves.

Missed the webcast but still have questions and comments? We’d love to keep the discussion going, so please leave your comments below!

This well-attended webinar generated a huge volume of questions from attendees, so we’ve decided to cut it into two parts. Stay tuned for the second segment tomorrow, but in the mean time, be sure to download and view the full webinar so you can join in!

Q: What are the kinds of cyber attacks that enterprises need to be aware of and who are the threat actors?

Richard Clarke (RC): It sounds like it’s a pretty fundamental question, but it’s confusing a lot of people because particularly the media are putting out all these stories about attacks and every week there’s another major enterprise that’s been attacked and it all gets mixed up in the blender like it’s all the same thing, and it’s not… I think it’s important that we distinguish among the actors and among the kinds of attacks because you can’t really respond to the sort of generalized idea of a hack, you have to respond to the specifics of who is attacking and how they are doing it. So the way I look at it is – I think there are four different kinds of phenomenon we are dealing with. The easy way to remember the four categories is the word CHEW, the first letter of each of the four types, Crime, Hacktivisim, Espionage, and at least potentially, Cyberwar.

Q: What do recent cyber attacks have in common?

RC: We see that there is a growing sophistication, attackers are using multiple techniques in the same attack, they’re using social engineering, vulnerabilities in client-side applications, vulnerabilities in web servers, and they’re doing two stage attacks, where there will be a precursor attack at a supplier company, things like that.

Q: So why are software applications at risk?

Chris Wysopal (CW): Your web applications, mobile applications, your software infrastructure, are parts of this chain of attacks.
RC: Well I think it boils down to the fact that it works. When your target is somebody like Sony or Citibank, which spends a lot of money on antivirus software, firewalls, intrusion detection, intrusion prevention, and even two-factor authentication, and maybe relies on certificates – how else are you going to get in? That’s your mission, that’s your target, that’s what you were told to get into, and you tried to do it the straightforward way, but you’re not going to get in so you keep trying and you eventually end up going in through the applications, or you go in through a third-party and go through their applications… The thing we don’t really traditionally think about is applications.

Q: What are the essential measures of software security that organizations need to be aware of?

RC: One of the things that should be on the list of essentials, is to verify third party code. If you don’t know what’s in the code, or if you’re just trusting the vendor, then you’ve got a problem because now you have no idea what they’ve failed to do, what their standards are, and how they’ve vetted it. There are lots of routine mistakes that people make when writing code, everybody does, and any code package, no matter how small, is going to have some of those mistakes. If they don’t have a systematic way of finding them, you’re in trouble.

Missed the webcast but still have questions? Keep the conversation going in the comments below!