I built a vulnerability scanner that refuses to lie to me

Hassan Balbakie — Mon, 24 Aug 2026 17:35:51 +0000

Static analysis tools have a trust problem. They read your source, pattern-match something that looks injectable, and hand you a warning — plus, increasingly, an AI-generated "fix." You have no idea if the warning is real or if the fix works. You just get more things to check.

I wanted the opposite: a tool that doesn't guess, doesn't trust the AI's word, and doesn't call anything fixed until it's proven fixed. So I built secfix.

The idea: proof by execution

Given a finding from Semgrep, secfix doesn't re-read the code. It runs it. It builds a test harness that calls the flagged function with a unique tainted marker, executes it inside a locked-down Docker sandbox, and records exactly where that marker ends up. If the marker lands unescaped inside a SQL string, or in a shell command, or in a file path that escapes its directory — that's a confirmed vulnerability, demonstrated by execution, not inferred from syntax.

Then it patches the function, runs the same harness again on a fresh trace, and only calls the fix validated if the marker now lands somewhere safe. Crucially: it never trusts the model's own description of what it changed. The AI's account of the fix is irrelevant — only a fresh execution trace counts. If re-verification doesn't come back clean, the patch is reported as unvalidated, full stop.

The whole thing is built to fail closed. When it can't prove something, it says uncertain — it never rounds up to "safe."

Where it broke — and why that's the interesting part

The tool passed all its tests. But passing your own fixtures proves very little. So I pointed it at real vulnerable Django apps.

It hit a wall immediately. Framework code isn't like a plain function — you can't just import a Django view and call it. The moment you touch it, Django wants settings, an app registry, a database, a request object. My harness imported the module and crashed before it reached a single line of the actual vulnerability.

I could have documented that as a limitation and moved on. Instead I went after it, one wall at a time: detecting the framework and its required Python version, matching the sandbox base image to it, calling django.setup(), baking a migrated database into the container, and constructing a real HTTP request with Django's own RequestFactory.

And it worked — I got a real Django view SQL-injection all the way to a confirmed verdict, the tainted input landing unparameterized in executed SQL, proven by running it.

The honest conclusion

But reaching that verdict took something I'm not going to pretend away: I had to hand-feed the tool knowledge it couldn't derive on its own — the specific database row the view expected, and a framework-internal file-size threshold buried in Django's upload handling. Some walls generalized cheaply. Others didn't — and the ones that didn't are exactly the ones that separate a clever solo project from the multi-year integration problem that funded security companies work on with whole teams.

I wrote all of it down — which walls fell easily, which didn't, and precisely where "automatic" stops. That map is the real deliverable. Not a demo where everything works, but an honest boundary of what execution-based vulnerability reproduction can and can't do on its own today.

The code, the safety design, and the full investigation are here: https://www.github.com/balbaks/secfix — with the Django deep-dive on the v1-django-bootstrap-spike branch.

I built a provenance protocol, audited my own auth, found 2 CRITICALs

Hassan Balbakie — Thu, 23 Jul 2026 10:45:50 +0000

I Built a Provenance Protocol, Audited My Own Auth, and Found 2 CRITICALs

I spent weeks building VERITAS — a protocol for verifiable provenance, identity, and attestation. 25 authenticated endpoints. Ed25519 signatures. Six layers: identity, content provenance, agent trust, economic escrow, governance.

Then I audited my own auth. It was broken.

CRITICAL #1: Every signed request was replayable forever

I had Ed25519 signatures on all mutating endpoints. That proves the sender controls the private key. What it doesn't prove is that the request is fresh. A captured (did, message, signature) tuple was valid indefinitely. Someone with access to a single logged request could replay it a thousand times.

The fix: a 60-second timestamp window. Every request now includes an ISO-8601 timestamp. The server rejects anything outside the window. A signature from two minutes ago is dead. Replay protection isn't a feature you add later — it's part of the signature verification itself.

CRITICAL #2: The server trusted caller-supplied messages

This one was worse. Every endpoint accepted a message field from the client, then verified the signature against it. The problem: the message the client signed and the parameters the server actually acted on were two different things. A client could sign "amount=10" while submitting amount=1000 in the body. The signature would verify — because it verified the message, not the operation.

The fix: canonical payload binding. The server now reconstructs the message from the actual request parameters it's about to act on. Keys are sorted lexicographically. Values are serialized with a deterministic float format. The signature is verified against the reconstructed message. If the body says amount=1000, the reconstructed message says amount=1000, and the signature must match that — not whatever the caller claims it should be. The message field was deleted from every request model entirely. That field was the vulnerability.

The implementation is in identity/did.py::canonical_message(). The critical line is the float serialization: f"{float(v):.8f}".rstrip("0").rstrip(".") — if your client uses str(v) or repr(v) instead, you get mystifying 403s. That one line is documented in the README because it's the thing that will bite anyone implementing a client.

The meta-test that prevents regression

Fixing bugs is easy. Proving they stay fixed is harder. I wrote a route-enumerating meta-test that programmatically discovers every authenticated endpoint, sends a request with a bad signature to each one, and asserts 403. If I ever add a new endpoint and forget to wire auth, the test fails. The two CRITICALs above can't silently return because the test suite won't let them.

18 tests total. 4 unit tests on the trust scoring engine. 14 integration tests covering auth rejection, replay protection, dispute resolution, and the full governance→arbiter→escrow→agent trust chain.

The demo that proves it works end-to-end

A six-second asciinema recording showing all six layers firing together:

An AI agent exceeds its 100-credit delegation cap
The cap is enforced at the agent endpoint (403)
The violation routes through escrow
A dispute is filed
Carol is elected arbiter through a real governance cycle
Carol resolves the dispute
The agent's trust score drops 50→40 because its side lost

Every step is a real HTTP call. The demo is a shell script in the repo. Anyone can clone and run it.

What I'd do differently

I shipped with no test suite. Don't. The first time Claude audited my code, it found 11 issues in one pass — missing auth on 20+ endpoints, private keys stored in the database, proof validation that counted instead of verifying. Every single one was fixed, and every fix got a test. But I should have written the tests first.

Repo

github.com/balbaks/veritas — please break it.

The threat model and known challenges are in docs/SPEC.md. The demo runs with bash scripts/demo.sh. If you find a vulnerability, open an issue. Public credit for every verified finding.

DEV Community: Hassan Balbakie