If it could have, why didn't it?

Anytime you see some someone describing how they used a new vulnerability research technique to find a bug, you’ll see an internet commenter state with confidence that static analysis could have found this bug. This pattern is so consistent as to be a law of physics. However, despite the internet commentariat’s confidence in static analysis, there are still a large number of vulnerabilities that are found by other techniques (including novel ones). The implicit premise behind these questions is that we do not need new vulnerability research techniques, or frequently that we don’t need new techniques for preventing the introduction of vulnerabilities (like memory safe programming languages), because you can achieve the same security outcomes already. I do not believe this is correct, and it’s worth examining why not.

As a starting point, it’s worth breaking down what people could plausibly mean by “static analysis could find this bug”. Internet comments are generally not written with this degree of precision, but it’s always worth understanding what people really mean.

The strongest version of that claim is “there exists a specific off-the shelf static analysis tool that, when run over this program, finds this exact bug”. If this is the claim, then you’re on solid grounds asking: why did no one run that static analysis tool? Of course, there’s also slightly weaker versions of this claim, for example that a piece of off the shelf static analysis software can be configured in some way to find this vulnerability, but doesn’t by default.

Another claim is something like, “existing static analysis tools can find similar bugs, so presumably they can be extended to find this as well”. Depending on the speaker’s expertise, the claim that existing analysis can be extended to this bug or pattern may be more or less well founded.

The weakest claim is something like, “surely it’s possible to write a static analysis that catches this”. Here the speaker isn’t even making a claim that there’s some off the shelf tool that can, would, or should find this bug, just that in principle it should be possible. Again, depending on the speaker’s expertise this might be true! And it’s entirely possible they’re right that the new static analysis would be valuable! But “you don’t need this new approach for finding vulnerabilities because some unknown person could [but hasn’t!] find it with a different approach” is not really very compelling.

However, focusing on the strongest version of the claim, we still have a pair of problems: false positives and sensitivity.

False positives are a well understood by practitioners, but are the kind of pesky little detail that too often get ignored. Simply put, if your static analysis correctly predicts 47 out of 5 vulnerabilities, teams will drown in triage and alert fatigue. In many many contexts, a tool that finds 3 out of 5 vulnerabilities with no false positives will be much more valuable to a security team. A tool that correctly identifies a vulnerability does not necessarily prevent that vulnerability in practice if it requires sorting through a mountain of false positives.

Of course, the converse problem exists as well: is your static analysis overly sensitive to implementation details and therefore miss basic variants of a pattern? An extremely common version of this is static analysis that works well when all the relevant logic is in a single function, but completely breaks down as soon as any of it is refactored into a separate function. Limitations like this are an extremely common reason that in theory an existing static analysis tool should have discovered a vulnerability, but in practice didn’t.

It’s also worth saying: it’s easy to say someone could write a static analysis that would find something specific after you’ve already found it! But if you want to claim the new technique is duplicative, you need to argue not just that static analysis can find this bug, but that someone would have had a reason to write this static analysis! Static analysis can generally only be used to hunt for known bug classes with many manifestations! There is almost always a trade-off between false positives and generality, the more specific to a given bug you make your proposed static analysis, the less plausible it becomes that someone should have known to implement the static analysis pass before the vulnerability was discovered by other means.

Finally, whenever someone claims their technique could have found a vulnerability, it’s worth asking: Oh, yeah? Why didn’t it? Particularly when we’re talking about vulnerabilities in widely known open source projects which are the subject of a lot of security research, it’s worth considering that maybe this vulnerability was not previously missed because no one had bothered to run some off the shelf static analysis tool, but in fact the new vulnerability research technique has real advantages. By the same token, if you want to demonstrate that a new technique is valuable, reporting net-new vulnerabilities is infinitely more compelling than re-finding known ones.