Skip to main content
February 27, 2007

Better Criteria for Selecting Pen Test Consultants

An article was forwarded to me today, entitled Avoid Wasting Money on Penetration Testing. While the core message is on target (i.e. be sure you know what you are getting before you sign on the dotted line), the suggestions for how to achieve this are misleading. Let's examine the "5 steps to choosing a supplier" outlined in the article:

Ask if their consultants have passed an independent penetration testing assessment. There are some services that will independently test a consultant and rate their strengths and weaknesses in great detail.

You are going to weed out a lot of top-notch pen testers if you create a gating factor based on some arbitrary certification. For one thing, the industry hasn't really reached consensus (or even close) on a meaningful pen testing certification. More importantly, it's usually large companies that encourage their consultants to pile up certifications like they're going out of style. This overemphasis on certifications is one of many reasons why the strongest pen testers typically gravitate toward security boutiques, which tend to place value on individual skillsets over industry certifications. Not to say that you won't find an expert pen tester in the Big 4 or a large security firm -- you certainly can, but they are just more scarce. It's very difficult to set the bar that high and still scale fast enough to keep up with the pipeline of a large consultancy. Bottom line, if you insist on a certification, you'll probably eliminate your best options right out of the gate.

Always meet the consultants doing the proposed assessment and satisfy yourself about their competence. Ensure that only the consultants interviewed are the one’s [sic] carrying out the assessments. It’s easy to wheel in a star consultant to win the business, but follow through with a trainee.

I don't know if the author meant that you should literally meet the consultants or whether a phone interview would suffice. I'll assume he meant the latter. On the surface, it's hard to find fault with this recommendation, but remember, you're dealing with consultants. Consultants by nature are used to sounding knowledgeable about topics they don't know very well. This will weed out some people but not others. Here's a perfect example -- trainers. These guys are incredibly polished when it comes to discussing attack vectors and their associated impacts, root causes, and remediation. They've delivered Application Security 101 twice a week for months on end, and they have their favorite case studies and analogies to illustrate each type of vulnerability in layman's terms. But they are usually not pen testers. 95% of them have never delivered a pen test in their life. One of these guys will impress the hell out of you on the phone but when it comes to sitting down at the keyboard and doing the pen test, they don't have the experience. I know, a trainer wouldn't be on the scoping call to begin with, but the point I'm trying to make here is that people can talk convincingly about pen testing without actually having the expertise to do it.

Ask to speak with reference sites, and actually follow through. Make sure the work carried out for those customers resembles your own.

Hard to go wrong by gathering references. However, the consultancy usually has certain pre-determined reference accounts to refer you to. It may not always be possible to get references for the specific consultants staffed on your pen test.

Ask questions about their methodology to discover if it is an active programme, or just marketing.

This is really just a variation on the second tip. It still boils down to the fact that being able to discuss the methodology intelligently doesn't imply expertise. It's nice to have a methodology, but having one doesn't necessarily mean that the consultants will follow it to a tee, as the author points out. Also, if you aren't a security expert yourself, how do you know whether or not the methodology is comprehensive? What does "comprehensive" really mean in the context of a black-box penetration test? Does the methodology take a checklist-style approach which is likely to consume the entire testing period, or does it build in time for veering off on tangents, where the most interesting and complex vulnerabilities are usually discovered.

Finally, remember that companies don’t perform penetration tests, people do. So no matter which company you go to, it always boils down to the person you have working on your account. Make sure you always have the best people for the job in place, and remember that the best person for one job, may not be the best for another. Understanding the strengths and weak-nesses [sic] of your team is a fundamental part of good management. Extending this principal [sic] to your suppliers is just as valuable.

Now that is a solid observation, one that I think is too often overlooked. It's easy to be swayed by the size and longevity of the company and forget that you should be picking the right one for the job. Remember, the bigger the consulting organization gets, the more likely the consultants will be generalists as opposed to specialists. Generalists are undeniably a better choice for certain jobs because of the additional perspective they bring to the table, but you don't want them delivering your pen test. Here are some additional things I would want to find out when picking a prospective vendor:

    • How specialized is the consultant? Do they deliver this particular service 20% of the time? 50%? 100%? You want to lean on the side of specialization. Trust me, there's a good reason they keep getting staffed on those projects.
    • How strictly do they adhere to the methodology? You want a good balance that establishes a "baseline" level of coverage but allows sufficient time for the consultant to try more creative attacks and follow the path of least resistance.
    • What tools does the consultant use? People who rattle off a laundry list of various tools may be overly reliant on that toolset and often lack the expertise to "roll their own" tools during the course of the testing.
    • Under what circumstances would they advise a customer to bear the risk of a vulnerability? If they can't give a good example of this, you might be dealing with someone who views security in a vacuum and doesn't consider other business factors when framing recommendations.

  Finally, word-of-mouth references are extremely valuable, and I don't mean the references supplied by the vendor. Talk to colleagues at other companies who have contracted similar work, and find out what they did and didn't like about the consultants they used? Did they feel that the technical assessment used a cookie-cutter approach, or was it customized to focus on attack vectors unique to the application? How was the communication during the project -- were there any surprises at the end, or did the delivery team keep the customer continuously informed on the findings? Don't forget the most obvious litmus test: would they hire this team again?

Chris Eng, Chief Research Officer, is responsible for integrating security expertise into Veracode’s technology. In addition to helping define and prioritize the security feature set of the Veracode service, he consults frequently with customers to discuss and advance their application security initiatives. With over 15 years of experience in application security, Chris brings a wealth of practical expertise to Veracode.

Love to learn about Application Security?

Get all the latest news, tips and articles delivered right to your inbox.