The shape of this post
This isn't a "look what I built" post. It's closer to a lab notebook — four small tools, shipped in sequence, each one built on the last, and each one deliberately documenting where it stops working rather than papering over it. If there's a thesis, it's this: a security tool's README is only as trustworthy as the test backing its weakest claim. So I tried to make every claim in every README point at a specific test that proves it.
1. secfix — the one that started it
The idea: most scanner findings are unverified guesses. Semgrep (or whatever) flags a line, and it's on you to figure out if it's actually exploitable. secfix closes that gap by executing the flagged code — building a pytest harness with a tainted sentinel value, running it in a locked-down Docker sandbox, and inspecting the execution trace (not the source) to decide confirmed / not_reproduced / uncertain. For confirmed findings, it generates a patch and re-runs the same harness on a fresh trace — it only calls something validated if that fresh trace proves the fix.
The single most valuable result in the whole project came from testing secfix against a real app: pygoat, a deliberately vulnerable Django app, scored 0 out of 84 findings reaching any verdict on the first pass.
That's not a failure, that's data. It mapped a real gradient of walls:
. DB-idiom detection missing common patterns like connection.cursor()
. Python/Django version mismatches breaking the sandbox
. Real vulnerabilities living in Django views that take request, not scalar parameters
. The hard wall: views doing DB lookups and permission checks before the vulnerable line, needing a seeded, migrated database and the exact right row to exist
The first three generalize with bounded engineering effort. The fourth — inferring a view's full data dependencies automatically — is genuinely team-scale work, the kind companies like Snyk spend years on. Getting one real Django view SQLi to a confirmed verdict required hand-supplying that knowledge. Worth doing once, to know exactly where the line is. Not worth faking past.
2. husk — pulling the sandbox out on its own
Buried inside secfix was a genuinely reusable piece: the locked-down Docker sandbox used to run flagged code safely. So I extracted it, generalized it, and gave it one job — detonate() untrusted Python, return stdout/stderr/exit code/timing, prove it's actually isolated.
The thing I wanted to fix here specifically: most sandbox READMEs list hardening flags (--network none, read-only rootfs, etc.) and never test whether an attacker running inside the container can actually violate them. So husk ships an adversarial test suite — scripts that actively try to open sockets, escape the filesystem, fork-bomb, and escalate privileges — and every claim in the README is tied to the specific test that proves it, checking the specific failure mode (a real Network is unreachable error, not just a nonzero exit code that could mean anything).
One design choice worth flagging: code is streamed into the container over stdin, never bind-mounted as a file. That removes the usual "which host paths are exposed" question entirely, rather than answering it carefully.
And stated above the fold, not hidden: husk shares the host kernel. It's Docker, not a hypervisor boundary — not a substitute for gVisor, Firecracker, or Kata against genuinely hostile code.
3. witness — from "did it work" to "what did it try"
husk tells you an attempt failed. It doesn't tell you what was attempted. witness closes that gap using Python's built-in audit-hook system (sys.addaudithook, PEP 578), which fires on operations like socket.connect, open, and subprocess.Popen before the sandbox's own isolation blocks them.
The mechanism reuses a trick from secfix: a unique per-run marker gets prepended to stderr output on every observed event, so the runner can cleanly split "structured behavior report" from "the script's actual stderr" afterward. Same sentinel-taint idea, different job.
Two walls came out of this branch, and only one was planned:
The intended wall: audit hooks only see Python-level API calls. Code that drops to ctypes and calls a raw syscall directly goes dark — the report can't see past that ctypes.dlopen call. So witness treats that call itself as a loud, explicit signal ("opaque escape hatch used, subsequent behavior not observable") instead of silently missing whatever happens next.
The wall I didn't expect: while wiring up the privilege-escalation category, os.setuid turned out to fire no audit event in CPython at all — not a bug in witness, a real fact about the interpreter, confirmed against CPython's own audit-events documentation. Rather than quietly drop that code path, I wrote a dedicated fixture and test proving it, and documented it as a second wall right next to the ctypes one. That's the kind of thing you only find by actually trying to build the feature and having it not work the way you assumed.
4. inlet — closing the loop on secfix's own Wall A
secfix's walls doc named a real, generalizable gap: DB-idiom detection missed patterns like connection.cursor(). inlet is that fix, built out as its own standalone static scanner: walk a Python codebase, find every call site that looks like SQL execution (raw DB API, Django .raw() / .extra(), SQLAlchemy text()), and classify each one as parameterized, concatenated, or uncertain.
The important thing inlet does not do: claim exploitability. A concatenated hit is a shape worth looking at, not a proven bug — pairing it with execution-based verification (like secfix) or manual review is still required. Saying that plainly, instead of implying more confidence than a static pass can honestly claim, is the whole point of the tool.
The wall here is structural: a single-function AST pass can't resolve a query string built in one function and passed into another — true resolution needs interprocedural dataflow analysis, which is its own project-scale undertaking. So instead of guessing, that case returns uncertain, and there's a dedicated fixture (builder_function_uncertain.py) and test proving it does exactly that rather than silently getting it wrong.
The pattern, stated plainly
Four tools, four walls, same discipline each time:
Write the thing that tries to break your own claim before you write the feature.
Let the tool fail loudly — uncertain, not_reproduced, opaque_escape_hatch — instead of rounding up to confidently wrong.
Put the limitation in the README above the fold, tied to the specific test that proves it, not as a footnote.
None of these four tools is trying to be the biggest thing in its category. Each one is trying to be exactly as trustworthy as its README claims to be — no more, no less.
Repos: github.com/balbaks/secfix · github.com/balbaks/husk · github.com/balbaks/witness · github.com/balbaks/inlet
Top comments (1)
README claims pointing at the test that proves them is the most honest documentation discipline I have seen for small tools. The failure mode it protects against is exactly the one I keep hitting in automation code: the README describes intent while the code describes reality, and only the drift between the two is interesting.
Did the wall show up more in testability (flaky network, external APIs) or in scope creep, where a "verified" claim needed infrastructure bigger than the tool itself?