Real world, how many here undergo in-depth security code reviews? Those that do, how often – once a quarter, once a version, once a blue moon? Those that don’t – why not? (Not referring to small or hobby programmers – not that I’m trivializing them, its just I don’t expect them to ;-) ).
As a security consultant, I’m usually the one called in to do the security reviews, however this is usually only either for very security-sensitive organizations (e.g. large banks, software vendors, military, etc), or as a result of regulatory requirements (e.g. PCI-DSS).
Now, few groups (except those in the biggest companies such as Microsoft, Intel, RSA, etc) really enjoy the review, even though it really should be a positive experience. It seems to me that this is mostly because of the perceived high investment – of resources, time, and of course cash to bring in the consultants.
Okay, so it’s not just perceived, it’s real enough: it’s commonly accepted that a single reviewer can cover between 50-100 LoC per hour. Though we’ve managed to multiply that – since we’re only looking for specific security issues (and the clients are pressing hard for lower costs) and we can minimize the scope according to the risk – we can max out at around 1000 LoC per hour. For any medium-to-large system, this is still hundreds of costly consultant hours, not trivial at all.
The common suggested alternative is automatic source code scanners, ala Fortify, ounce labs, etc. However, besides the licensing costs, this is far from efficient – typically we find these tools to produce results in the 100 thousands, with a very high (70-90%) rate of false positives (and duplicates). So you’re still spending large chunks of time going over the results, AND these tools do not cover a substantial set of potential vulnerabilities (e.g. logic flaws, business logic, etc)
That said, (and a big DISCLAIMER should go here:) I’ve been working the past few months with one of the tool vendors to develop a service that would do this very efficiently – e.g. be able to cover 500K LoC in just around a single week of work, and yet provide actual, real accurate and complete results – virtually ZERO false positives and nearly no missed false negatives.
Those of you who should be doing SCR, but aren’t – would this be enough to convince you otherwise? or is there something else holding you back? Or is it just not an issue for you?
To clarify, I’m not trying to promote myself or my service, just trying to get some real-world perspective beyond my own security-evangelistic agenda. I’d like to see the issues as other programmers see them…
Further clarification, I am NOT asking HOW to perform code reviews, what options there are, etc. I have much expertise in this field, and it is this expertise that I sell to my clients. My question is IF code reviews are as unpopular as they seem, WHY this specific activity is not as popular as it should be, and HOW we can go about changing this. (Irrelevant of choice of methodology, tool, etc.)
Furthermore, as Corneliu and others pointed out, security code reviews should NOT be taken alone as the sole protection and verification of a system’s security, rather should be one element of a complete, holistic SDLC (secure development lifecycle) framework. However, neither should it be forgotten. So my question is really focusing on that one element, whether in the context of the full SDLC or alone as a step forward from penetrate-and-patch.