I was just reading Dre's post, R.I.P. CISSP, over at the tssci security blog, in which he predicts the upcoming OWASP People Certification Project will be the next big thing. This paragraph is quoted from James McGovern's blog (James is the project leader):

As an Enterprise Architect, I understand the importance of the ability for a security professional to articulate risk to IT and business executives, yet I am also equally passionate that security professionals should also have the capability to sit down at a keyboard and actually do something as opposed to just talking about [it].

I agree wholeheartedly with this sentiment, and I believe the project goals are noble. So I went to read the latest OPCP draft proposal to see how they planned to tackle this admittedly difficult problem. What did I find? It's just another test, with questions in a dozen or so broad categories. Far more specialized that CISSP, with topics that are more relevant to application security, but ultimately, still just a test.

The comment I once made about security educators/trainers is relevant here. Whatever questions end up on the OPCP test, these educators could probably answer most of them correctly without even studying. They lecture day in and day out about these topics. They have heard obscure questions and are prepared to answer them. And yet, many of them do not have any practical field experience.

A client chastised me once for making a statement that penetration testing is a mixture of art and science. He wanted to believe that it was completely scientific and could be distilled down to a checklist type approach. I explained that while much of it can be done methodically, there is a certain amount of skill and intuition that only comes from practical experience. You learn to recognize that "gut feel" when something is amiss. He became rather incensed and, in effect, told me I was full of it. This customer went on to institute a rigid, mechanical internal process for web app pen testing that was highly inefficient and, ultimately, still relied mostly on a couple bright people on the team who were in tune with both the art and the science.

Certifications only test the science.

Veracode Security Solutions


Security Threat Guides

About Chris Eng

Chris Eng, vice president of research, is responsible for integrating security expertise into Veracode’s technology. In addition to helping define and prioritize the security feature set of the Veracode service, he consults frequently with customers to discuss and advance their application security initiatives. With over 15 years of experience in application security, Chris brings a wealth of practical expertise to Veracode.

Comments (6)

Zach | June 20, 2008 4:34 pm

Dig the post, Chris. It's comforting to know that the argument for the art+science aspect is shared.

At my work, we've run into the problem time and time again of trying to standardize and sort of rigidify our testing methodologies, only to come back to leaving a great degree of wiggle room for the more visceral element of it all. For most types of engagements, we elected to create high(er) level "objectives" that need to be achieved or touched upon, allowing for a better flowing, more adaptable test.

Andre Gironda | June 20, 2008 7:11 pm

Chris, great post.

I tend to think that the OPCP exam questions will include authors from many different backgrounds, including those with real world field experience.

In the world of software quality, there have been authors that have attempted to harness in on the artistic aspects that you speak of into their mostly-science based books and research papers.

For example, in books by Cem Kaner (Testing Computer Software) and Lee Copeland (A Practitioner's Guide to Software Test Design) -- the authors identify testing paradigms such as those that are scripted (as in Bourne Identity the "movie script" kind, not Bourne shell the "Unix script" kind) vs. those that are exploratory.

I would tend to think that having knowledge about what exploratory testing is, how it works, what the advantages/disadvantages are, and how they apply to certain test cases (e.g. Allpairs black-box testing and its use of "tester's choice", state-transition testing, or combinatorial explosions) would most certainly be a part of the OWASP exam questions. Combinatorics is especially important for web application security testing and development -- RSnake and many others have accounted for the "Death by a Thousand Cuts" concept. Even my other recent post on "What web application security really is" speaks to the fact that the focus of web application attacks is not the classic solitary critical bug, but a string of smaller bugs that when stringed together form a much more complex and interesting attack. Also see: The MySpace worm (XSS+CSRF+XHR), Ajax Security by Billy Hoffman and Bryan Sullivan, et al.

It's fairly well-known common knowledge that testing of business logic for web applications require at least two accounts at every defined access level (e.g. user, administrator, etc), and that this goes beyond authorization testing to include various things such as object passing through parameter tampering (including cookie tampering). Surely you can think of others? Pair testing? Testing reputations? The Think-Aloud Protocol? Screen captures and desktop sharing?

Having identified severe vulnerabilities in web applications such as Google, eBay, and countless others -- I can certainly affirm that the gut feeling accounts for quite more than an inexperienced person would allow for. Sometimes you just see things and they stick out like a sore thumb.

CEng | June 20, 2008 8:09 pm


Yeah, it is hard to run a security practice without methodologies. For example, they are a good way to ensure consistency and make sure that every pen test covers certain elements (i.e. avoid the "oops, I forgot to check for CSRF" situations). When I was doing security consulting, we always made sure that our methodology carved out time for the consultant to use at his/her own discretion, specifically for those situations requiring deeper analysis that can't be scoped in advance.


I have no doubt that the OPCP will receive contributions from plenty of people with significant field experience. But I think that regardless of who the contributors are, it's nearly impossible to prepare a written test that measures hands-on abilities. I think you need a hands-on test for that. The trouble with that is that it's orders of magnitude more complex to maintain.

Let's say you write a vulnerable application and require people to pen test it. Or you give them a sniffer and ask them to do some traffic analysis and deconstruct an unknown protocol, then fuzz it. Or a similar task. You'd spend a fair amount of time building the test environment, documenting where the problems were, and deciding how the scoring would work. 100 people take the test. The minute they're done, the test material is no longer useful because everyone will just go tell their friends and the next group sitting for the test has an unfair advantage. This advantage grows as the test materials get increasingly stale.

With a multiple choice test, you can recycle a lot of the same questions, swapping the choices around or adding new "wrong" answers. With a hands-on test, there's a lot more work involved in creating fresh content, but how else can a certification truly measure hands-on abilities?

I think that the OPCP has the right idea in terms of focusing the subject matter on relevant areas. But if we are going to have a certification for security practitioners that really means anything, it's got to have a hands-on component. A written test only scratches the surface, regardless of who is contributing the questions.

Kurt Grutzmacher | June 22, 2008 10:57 pm

Without a doubt penetration testing is more art than science but within every art form there is a fundamental structure involved and that's what tests like the OPCP and the CISSP try to cover. As a tester I fully believe more in the "artistic" form when performing a test but it's always tough to answer the question "so what will you be doing during the test, can you give us a test script?" when talking with a project manager.

I always wanted to answer in my best snobby-artist tone: "One can not put a confining box around such creativity! One has to be ONE with the browser in order to achieve great results!"

If you want to look at a test with a real-world component, take a look at the CCIE exams. Before the revamped it 6 or so years ago you had to first pass the written in order to qualify for the TWO DAY build / break / fix hands-on lab. Given the breadth of networking options it wasn't unlikely to be given such gems as "Implement ATM LANE, Frame Relay backbone, multiple BGP hops, RIPv1 to IGRP, AppleTalk, Banyan VINES and an IPX-based dial-up" by noon on day 2. Oh and you're lab equipment isn't cabled up at the start and you need to recover the password from these two Cisco 4000 routers. After lunch on day 2 (if you make it that far) the lab techs will screw up your environment and you have a few hours to fix it. Good luck!

Such a test was very time consuming for Cisco and costly to develop and administer but for the longest time the CCIE was the pinnacle of network certifications because of it and, in return, nearly everyone used Cisco gear. A similar thing COULD be created for Web App security but who would administer it, who would spend the time taking it and would it really help employers find suitable people for the money they would expect after having passed such a grueling thing? There's no incentive for any one company to do this and if its done by committee then there's little hope in it being the "best of breed."

Certs can help when finding good candidates for a penetration tester position but finding the right person is very tough. The one thing I always look for is that "sparkle" that says "I know don't everything but I like knowing about everything." The last time I had a position to fill it took nearly a year of wading through crappy resumes, inept interviewees and people who "knew nmap" before finding the right candidate.

Isaac Dawson | June 23, 2008 1:18 am

Hi Chris,
Long time no chat ;>. Boy, I couldn't agree more on the gut feel issue. The past few years of me being in Japan training people I've been in far too many situations where both consultants and the customer's ask me how I found a specific issue. It really is hard saying it was by gut feel; "oh I just figured if I changed the hidden variable from Japanese to English that error handling may not be sufficient and allow me to find another serious problem." From the perspective of a customer, I can totally understand why they get frustrated with this sort of response. But at the same time there's really no other way to describe it. Some people got it, others do not. Back to the point of written tests/hands on. I think one way they could enhance these sorts of tests if they have written essays on a proposed problem; why do you think its a problem? how you would attack it? What you think is happening behind the scenes? This gives the reviewer an excellent look into how the tester understands the problem, approaches a problem and ultimately how they think about why the problem exists. Obviously without seeing source you can not be 100% sure, but I think this is much better than multiple choice. The obvious downside is you now need to pay reviewers who actually understand security instead of feeding the answers into an automated system ;>.

CEng | June 24, 2008 8:57 pm


I've not had any direct experience with the CCIE but have heard some war stories similar to yours. It may be the only certification I'm aware of that really puts candidates through their paces.

Agree that there must be fundamental structure to any pen test engagement; see my earlier comment to Zach.


Wow, nice to hear from you! Interesting idea on the essay questions. Certainly a better way that multiple choice to measure how well somebody understands a concept. Creating and maintaining the content would be tough but not as difficult as writing a new application every time the test was administered.

Please Post Your Comments & Reviews

Your email address will not be published. Required fields are marked *

Love to learn about Application Security?

Get all the latest news, tips and articles delivered right to your inbox.