Frameworks are not auto-hackers

I was reading a review of the Yoggie Gatekeeper Pro in this month's SC Magazine. It's a neat little device that hides your PC behind a Linux firewall-appliance when connecting to an untrusted network. The voodoo of how it shims itself into your Windows networking stack so you can connect to a wireless network and still be protected through the Yoggie aside -- one thing about the review really made my hair bristle:

Using our vulnerability assessment tool (NetClarity) and our penetration tool (Core Impact) we were unable to compromisethe Gatekeeper or the computer behind it.
- SC Magazine, April 2007, Pg 63

Well duh.

Both the tools listed are only as strong as their signatures, exploits and platform shellcode. That statement is like running Core Impact against a copy of OpenVMS and saying IMPENETRABLE! when you're done. Technically it's valid but it's no measure of strength.

Maybe these statements are made because of a contractual obligation. "Say our product name five times and we'll give you free copies" sort of thing. Unfortunately there will be InfoSec managers and the like who will listen and wonder if maybe they should use these tools in lieu of hiring security professionals who actually know something.

Maybe I'm just being too overly critical and hypersensitive about this. I don't think I am as I've looked at a number of Web Application Security tools on the market and none of them have been able to find the more serious vulnerabilities vs. a team of two or three highly skilled testers have. We still need good QA but attack Frameworks like CORE Impact, Canvas and Metasploit aren't automated tools. Don't treat them as such.