I've always been a fan of Joel Spolsky's Guerrilla Guide to Interviewing. Unfortunately, I've never been able to apply it in its purest form because in recent years, I've been hiring mostly application security consultants, not software engineers. However, the structure is still remarkably useful, with some modifications. So, without further ado, here's an example of how one might apply Guerrilla Guide techniques when interviewing a candidate for an application penetration testing position.
Sample Question #1: What is the difference between a GET and a POST?
Note that I'm not violating the Guerrilla Guide's "no trivia questions" rule by asking about some arcane feature of the HTTP protocol that only the RFC author would know. All I know is, anybody who can't answer this question is a No Hire. If they can answer it, see how deep they go -- there's one very obvious answer, but it's also a relatively open-ended question.
Sample Question #2: Explain how cookies work.
You'd expect to hear some discussion of the HTTP protocol here. If the person starts talking in API abstractions and can't describe what's happening one level down, we're in trouble. Otherwise, you want to see how much detail they can provide beyond the basic answer.
Sample Question #3: Let's say you wanted to brute force a POST request to http://example.com/foo, iterating the parameter 'bar' over a numeric range. How would you accomplish it?
This fulfills the "Easy Programming Question" objective of the Guerrilla Guide and is a good way of seeing how the candidate will approach a relatively straightforward task. Most skilled pen testers I've worked with would do this from the command line or write a quick script. You're not looking for a particular scripting language, you just want to see that it's familiar enough that you can quickly churn out the few lines of code. If the candidate struggles with it, or even worse, doesn't understand what you're asking for, that's a bad sign.
Sample Question #4: Describe one of your most complex or unique attacks against a web application.
This is a good open-ended question and fulfills the Guerilla Guide's "Recent Project" objective. Chances are, if the candidate has done more than a couple dozen pen tests, there was at least one time where they found and exploited something that required deep analysis and/or creative thinking. If they can't come up with anything, you're dealing with a novice. If they do, ask follow-on questions to make sure that they really understand the attack and can explain it in a coherent fashion. If they can explain the outcome of the attack but nothing else, there's a good chance they either don't communicate well or are trying to pass off someone else's anecdote as their own. Finally, step back and ask yourself what was creative or complex about this particular finding. Remember, you asked for their best. If it's something commonplace (e.g. "I used a SQL injection vuln to own the box with xp_cmdshell"), you've now established a baseline of this candidate's abilities.
Wrapping It Up
Beyond these specific questions, you might also want to see how well the candidate can explain a common web application vulnerabilities, from root cause to exploitation methods to remediation. This is extremely important if you're looking to hire a consultant, less important if it's not a customer-facing role.
You can also ask their opinion on current application security topics or events. It's more important that they actually have an opinion (and can back it up) than whether or not you agree with them. Ultimately, however, the fundamental tenet of the Guerrilla Guide still applies -- "When you find the smart, gets-things-done candidate, you'll know it. If you're not thrilled with someone, move on."
For all of you web app security people out there -- what's your favorite interview question?