An article was forwarded to me today, entitled Avoid Wasting Money on Penetration Testing. While the core message is on target (i.e. be sure you know what you are getting before you sign on the dotted line), the suggestions for how to achieve this are misleading. Let's examine the "5 steps to choosing a supplier" outlined in the article:

Ask if their consultants have passed an independent penetration testing assessment. There are some services that will independently test a consultant and rate their strengths and weaknesses in great detail.

You are going to weed out a lot of top-notch pen testers if you create a gating factor based on some arbitrary certification. For one thing, the industry hasn't really reached consensus (or even close) on a meaningful pen testing certification. More importantly, it's usually large companies that encourage their consultants to pile up certifications like they're going out of style. This overemphasis on certifications is one of many reasons why the strongest pen testers typically gravitate toward security boutiques, which tend to place value on individual skillsets over industry certifications. Not to say that you won't find an expert pen tester in the Big 4 or a large security firm -- you certainly can, but they are just more scarce. It's very difficult to set the bar that high and still scale fast enough to keep up with the pipeline of a large consultancy. Bottom line, if you insist on a certification, you'll probably eliminate your best options right out of the gate.

Always meet the consultants doing the proposed assessment and satisfy yourself about their competence. Ensure that only the consultants interviewed are the one’s [sic] carrying out the assessments. It’s easy to wheel in a star consultant to win the business, but follow through with a trainee.

I don't know if the author meant that you should literally meet the consultants or whether a phone interview would suffice. I'll assume he meant the latter. On the surface, it's hard to find fault with this recommendation, but remember, you're dealing with consultants. Consultants by nature are used to sounding knowledgeable about topics they don't know very well. This will weed out some people but not others. Here's a perfect example -- trainers. These guys are incredibly polished when it comes to discussing attack vectors and their associated impacts, root causes, and remediation. They've delivered Application Security 101 twice a week for months on end, and they have their favorite case studies and analogies to illustrate each type of vulnerability in layman's terms. But they are usually not pen testers. 95% of them have never delivered a pen test in their life. One of these guys will impress the hell out of you on the phone but when it comes to sitting down at the keyboard and doing the pen test, they don't have the experience. I know, a trainer wouldn't be on the scoping call to begin with, but the point I'm trying to make here is that people can talk convincingly about pen testing without actually having the expertise to do it.

Ask to speak with reference sites, and actually follow through. Make sure the work carried out for those customers resembles your own.

Hard to go wrong by gathering references. However, the consultancy usually has certain pre-determined reference accounts to refer you to. It may not always be possible to get references for the specific consultants staffed on your pen test.

Ask questions about their methodology to discover if it is an active programme, or just marketing.

This is really just a variation on the second tip. It still boils down to the fact that being able to discuss the methodology intelligently doesn't imply expertise. It's nice to have a methodology, but having one doesn't necessarily mean that the consultants will follow it to a tee, as the author points out. Also, if you aren't a security expert yourself, how do you know whether or not the methodology is comprehensive? What does "comprehensive" really mean in the context of a black-box penetration test? Does the methodology take a checklist-style approach which is likely to consume the entire testing period, or does it build in time for veering off on tangents, where the most interesting and complex vulnerabilities are usually discovered.

Finally, remember that companies don’t perform penetration tests, people do. So no matter which company you go to, it always boils down to the person you have working on your account. Make sure you always have the best people for the job in place, and remember that the best person for one job, may not be the best for another. Understanding the strengths and weak-nesses [sic] of your team is a fundamental part of good management. Extending this principal [sic] to your suppliers is just as valuable.

Now that is a solid observation, one that I think is too often overlooked. It's easy to be swayed by the size and longevity of the company and forget that you should be picking the right one for the job. Remember, the bigger the consulting organization gets, the more likely the consultants will be generalists as opposed to specialists. Generalists are undeniably a better choice for certain jobs because of the additional perspective they bring to the table, but you don't want them delivering your pen test. Here are some additional things I would want to find out when picking a prospective vendor:

    • How specialized is the consultant? Do they deliver this particular service 20% of the time? 50%? 100%? You want to lean on the side of specialization. Trust me, there's a good reason they keep getting staffed on those projects.
    • How strictly do they adhere to the methodology? You want a good balance that establishes a "baseline" level of coverage but allows sufficient time for the consultant to try more creative attacks and follow the path of least resistance.
    • What tools does the consultant use? People who rattle off a laundry list of various tools may be overly reliant on that toolset and often lack the expertise to "roll their own" tools during the course of the testing.
    • Under what circumstances would they advise a customer to bear the risk of a vulnerability? If they can't give a good example of this, you might be dealing with someone who views security in a vacuum and doesn't consider other business factors when framing recommendations.

  Finally, word-of-mouth references are extremely valuable, and I don't mean the references supplied by the vendor. Talk to colleagues at other companies who have contracted similar work, and find out what they did and didn't like about the consultants they used? Did they feel that the technical assessment used a cookie-cutter approach, or was it customized to focus on attack vectors unique to the application? How was the communication during the project -- were there any surprises at the end, or did the delivery team keep the customer continuously informed on the findings? Don't forget the most obvious litmus test: would they hire this team again?

About Chris Eng

Chris Eng, vice president of research, is responsible for integrating security expertise into Veracode’s technology. In addition to helping define and prioritize the security feature set of the Veracode service, he consults frequently with customers to discuss and advance their application security initiatives. With over 15 years of experience in application security, Chris brings a wealth of practical expertise to Veracode.

Comments (3)

Chris | March 2, 2007 8:32 am

Good post. Unfortunately the market is saturated with certifications and many 'penetration testers' that arent qualified. The more automated our toolsets become you will find less knowledgeable penetration testers. This is sort of evident when you look back at the explosion of quality low cost scanners (Nessus, Retina etc...) that took off 5 or 6 years ago. Metasploit is great, but a majority of the penetration testers out there don't understand it. Its easier to plug an IP into Nessus and hit 'Start Scan'.

CEng | March 2, 2007 11:03 am

This is true. Add this to my list: "Make sure your definition of a penetration test is consistent with the vendor's definition." Too many consulting companies these days try to pass off a few automated scans (and weeding out of the false positives) as a penetration test. Sorry guys, but that's a vulnerability scan and nothing more. The last thing you want to do is sign on the dotted line without being aligned on expectations.

Don't get me wrong, automation has its place. Say you're responsible for the security of 500 web applications and you want to scan them on a rotating basis to find all the low-hanging fruit. You'll undoubtedly find some explotiable vulnerabilities that need fixing. Just don't get a false sense of security, because all the automated scanners do is fling junk at every parameter of every page and use heuristics to gauge the response. They find a lot of problems, but they won't even touch higher-level logic issues, authorization bypass, etc. It's not a pen test, it's a scan.

The network side is different. Metaploit, CANVAS, Core Impact, etc. give you the tools you need to gain and extend access. Nessus, Retina, etc. are just scanners.

sandrar | September 10, 2009 8:23 am

Hi! I was surfing and found your blog post... nice! I love your blog. :) Cheers! Sandra. R.

Please Post Your Comments & Reviews

Your email address will not be published. Required fields are marked *

Love to learn about Application Security?

Get all the latest news, tips and articles delivered right to your inbox.