A few weeks ago I ran inlet, a small static SQLi scanner I built, against 15 real Python packages — 5 with documented CVEs, 10 without — and reported the result plainly: 0 clean hits on the 5 CVEs, 1 partial. The obvious next question was whether an established tool would've done better on the same corpus. So I ran it.
The setup
Same 15 packages, same versions, same CVE fix-commits used as ground truth. Semgrep, using its current public Python security rulesets. No changes to either tool — this was a reference run, not a tuning exercise.
The result that actually surprised me
Semgrep missed the exact same 4 CVEs inlet missed. Not similar ones — the same four, and for the same reason: the vulnerable line is reached through a framework abstraction (a hook method, a query-builder helper) that never literally calls anything either tool's rules recognize as a SQL-execution sink. Neither tool's idiom coverage extends that far.
This is a more interesting result than either "Semgrep obviously wins" or "inlet holds its own" would have been, because of what it rules out. Before this benchmark, inlet's 4/5 miss rate had an easy, dismissive explanation available: small idiom list, one week of work, of course it misses things a mature community-maintained tool wouldn't. This result removes that explanation. If years of accumulated Semgrep rules also miss these exact four, for the exact same mechanism, the ceiling isn't "inlet needs more rules" — it's a real limit on what name-based static matching can see at all, regardless of how many names you add to the list.
Where they actually differed
Semgrep caught one CVE (Archery's) that inlet only partially got — the vulnerable f-string assignment sat inside a try: block, a name-resolution gap in inlet's local-scope tracing that Semgrep's rule engine handled better.
On the 10 non-CVE packages, inlet flagged real risk shapes in a few (records, dataset, SQLModel, Flask-SQLAlchemy) that Semgrep's shape-based rules structurally don't attempt — largely because inlet has an explicit uncertain verdict for "I can't resolve this argument," while a lot of static tools either flag confidently or stay silent, with nothing in between.
What this means, stated plainly
Neither tool dominates the other on this corpus. That's the honest outcome, and it's more useful than a clean win would have been — it turns inlet's original 0/5 from "here's a limitation of my implementation" into "here's evidence about a limitation of the approach itself," which is a meaningfully stronger claim, and one I couldn't make before running this comparison.
Full results, per-package notes, and the fairness caveats (Semgrep's multi-year community maintenance vs. inlet's one-week solo build, judgment calls in rule selection) are in EVALUATION.md:
github.com/balbaks/inlet
Top comments (0)