Comments on: Missing the Point http://www.veracode.com/blog/2008/07/missing-the-point/ Application security testing, analysis, and metrics Tue, 15 May 2012 22:16:53 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: Chris W. http://www.veracode.com/blog/2008/07/missing-the-point/comment-page-1/#comment-1971 Chris W. Wed, 23 Jul 2008 15:39:49 +0000 http://www.veracode.com/blog/?p=121#comment-1971 Veracode recommends that high and very high assurance applications, the majority of online apps that handle PII and financial data, also get a manual review in addition to an automated analysis. You can see this on page 2 of our software rating system datasheet: http://www.veracode.com/images/pdf/datasheet_application_security_ratings_system.pdf Veracode recommends that high and very high assurance applications, the majority of online apps that handle PII and financial data, also get a manual review in addition to an automated analysis.

You can see this on page 2 of our software rating system datasheet:
http://www.veracode.com/images/pdf/datasheet_application_security_ratings_system.pdf

]]> By: OWASP http://www.veracode.com/blog/2008/07/missing-the-point/comment-page-1/#comment-1970 OWASP Wed, 23 Jul 2008 01:43:12 +0000 http://www.veracode.com/blog/?p=121#comment-1970 Just pointing people to a good reference. Glad to hear you confirm that you don't find these critical security areas. Many tool vendors aren't very realistic about what their tools can't do. I was probably thrown off by your PCI compliance document which says, "Additionally, using automated dynamic and binary analysis, Veracode looks for the OWASP top ten, the NIST list as well as many other business best practices" This seems to be the exception for you all, but there are a huge number of vendors who seem hell bent on overstating their abilities, claiming to find all violations of the OWASP Top Ten. Claiming all 695 issues in CWE might be a record though. Just pointing people to a good reference. Glad to hear you confirm that you don’t find these critical security areas. Many tool vendors aren’t very realistic about what their tools can’t do.

I was probably thrown off by your PCI compliance document which says, “Additionally, using automated dynamic and binary analysis, Veracode looks for the OWASP top ten, the NIST list as well as many other business best practices”

This seems to be the exception for you all, but there are a huge number of vendors who seem hell bent on overstating their abilities, claiming to find all violations of the OWASP Top Ten. Claiming all 695 issues in CWE might be a record though.

]]> By: Chris Eng http://www.veracode.com/blog/2008/07/missing-the-point/comment-page-1/#comment-1964 Chris Eng Tue, 22 Jul 2008 03:50:11 +0000 http://www.veracode.com/blog/?p=121#comment-1964 @OWASP (do you really speak for the entire OWASP?): Most automated tools don't claim to detect authentication, access control, encryption, or business logic vulnerabilities. I don't think any of them claim to be a complete solution. A human can certainly find more complicated problems, but he isn't going to find all 500 XSS vulnerabilities. He's going to find 5 representative examples before moving on to more interesting attack vectors, and guess what? The developer is only going to fix the 5 examples he was given. Automated analysis and manual analysis are complementary. Nobody should expect automation to be 100% accurate -- it's a hard problem to solve. But even with imperfections, it's still valuable. Manual testing has different imperfections, not the least of which is that results quality is dependent on the skill of the tester. The old adage still holds true -- "A fool with a tool is still a fool." @OWASP (do you really speak for the entire OWASP?):

Most automated tools don’t claim to detect authentication, access control, encryption, or business logic vulnerabilities. I don’t think any of them claim to be a complete solution.

A human can certainly find more complicated problems, but he isn’t going to find all 500 XSS vulnerabilities. He’s going to find 5 representative examples before moving on to more interesting attack vectors, and guess what? The developer is only going to fix the 5 examples he was given.

Automated analysis and manual analysis are complementary. Nobody should expect automation to be 100% accurate — it’s a hard problem to solve. But even with imperfections, it’s still valuable. Manual testing has different imperfections, not the least of which is that results quality is dependent on the skill of the tester.

The old adage still holds true — “A fool with a tool is still a fool.”

]]> By: OWASP http://www.veracode.com/blog/2008/07/missing-the-point/comment-page-1/#comment-1963 OWASP Tue, 22 Jul 2008 02:42:31 +0000 http://www.veracode.com/blog/?p=121#comment-1963 But automation *is* dangerous. People are *relying* on these tools because they've been sold as a cure. But the fact is that they're blind to the majority of application security problems. They can't find most authentication, access control, encryption, or business logic vulnerabilities. And what they do find is riddled with false alarms that take a huge amount of time to diagnose. And even if they do find a real issue, they can't evaluate the business impact of the problem. * Working on the wrong stuff first is dangerous. * Wasting critical application security resources is dangerous. * Operating with a false sense of security is dangerous. Still, just because a chainsaw is dangerous doesn't mean it's useless. Automated application security tools can be helpful to an experienced security expert. They may be able to provide some data as input to a security review. But automation *is* dangerous. People are *relying* on these tools because they’ve been sold as a cure. But the fact is that they’re blind to the majority of application security problems. They can’t find most authentication, access control, encryption, or business logic vulnerabilities. And what they do find is riddled with false alarms that take a huge amount of time to diagnose. And even if they do find a real issue, they can’t evaluate the business impact of the problem.

* Working on the wrong stuff first is dangerous.
* Wasting critical application security resources is dangerous.
* Operating with a false sense of security is dangerous.

Still, just because a chainsaw is dangerous doesn’t mean it’s useless. Automated application security tools can be helpful to an experienced security expert. They may be able to provide some data as input to a security review.

]]>