Cloudflare published the skill that seeded their vulnerability discovery harness, and it is a good skill: six phases, adversarial validation, structured verdicts, and an explicit instruction to report what you could not verify instead of guessing. I ran it on my own repository and it found real bugs.
It did not run out of the box. Not because the skill is wrong, but because it assumes three things about the environment that a normal developer setup does not have: an agent that can spawn isolated parallel subagents, an OS-enforced sandbox for executing target code, and Node available on the parent side to validate the artifacts. I had a laptop-grade agent CLI and no sandbox. Most of the work was building those three things, and that work is reusable.
What the skill actually requires
Straight from the repository, the requirements are:
- a coding agent whose model supports tool use and parallel sub-agents
- Node.js, for the two zero-dependency validators that check the coverage ledger and the findings file against their schemas
- an OS-enforced sandbox that disables external networking, uses a sanitized allowlisted environment, enforces resource limits, and permits writes only to assigned scratch paths
The last one is not optional advice. The skill says that without all of those controls it will not execute target code, and the affected findings stay at needs_validation. That is the correct behaviour: an audit that runs a target's test suite unsandboxed is a supply chain incident with extra steps.
The pipeline itself is six phases: reconnaissance into an architecture map and a coverage ledger, coverage-led hunting with isolated hunters, candidate validation by a fresh verifier, structured output to findings.json, independent verification of the final source claims, and reporting derived from the verified records. Verdicts are confirmed, needs_validation, and rejected, and the skill is explicit that a rejected candidate is a disproved candidate, not a harmless one.
Gap 1: the agent runtime
The skill delegates. A single agent doing all the hunting sequentially would take days, and worse, it would review its own work. So the first question was which CLI could actually spawn isolated subagents with a skill attached and permission restrictions.
| Runtime | Result |
|---|---|
| pi | no task tool, so no subagents. Cannot run this skill |
| Command Code | has a task tool, but the Go quota throttled runs into stopping partway |
| opencode (V1) | skill tool, task tool and permission maps all present. Completed all three runs |
The detail that made opencode work is the role split. I defined two subagent roles as files: security-hunter and security-verifier. Both are read-only on the code, and the verifier is configured with task: false so it cannot spawn children of its own. That is what makes verification independent: the agent that checks a finding has no way to delegate the checking back to the agent that produced it.
Gap 2: the sandbox
I tried bubblewrap first, because it is the lightest option and it needs no daemon. It failed immediately on Ubuntu with loopback: Failed RTM_NEWADDR. The cause is a kernel setting that restricts unprivileged user namespaces (kernel.apparmor_restrict_unprivileged_userns=1), which turns bwrap into a tool that cannot set up its own network namespace.
Docker worked, with three corrections that cost me a round each:
-
--tmpfsmounts arenoexecby default. The build tools inside the scratch directory then fail with permission errors that look like compilation problems. It needsexecin the mount options. -
debian:bookworm-slimdoes not shipsqlite3. The target's test suite had database tests, and about a third of them failed for a reason unrelated to security. Adding the package fixed the noise. -
GOTOOLCHAIN=autotries to download a toolchain, which cannot work when the container has no network. PinningGOTOOLCHAIN=localand bind-mounting aGOROOTthat matches the module's Go directive was the fix. Dependencies were vendored on the host before the snapshot, so the container never needs to fetch.
The resulting invocation, trimmed to the parts that matter:
docker run --rm --network none --read-only \
--tmpfs /scratch:rw,exec,size=2g,mode=1777 \
--mount type=bind,src=$SNAPSHOT,dst=/target,readonly \
--mount type=bind,src=$TOOLCHAIN,dst=/toolchain,readonly \
-e GOROOT=/toolchain -e PATH=/toolchain/bin:/usr/local/bin:/usr/bin:/bin \
-e HOME=/scratch -e GOTOOLCHAIN=local -e GOFLAGS=-mod=vendor -e CGO_ENABLED=0 \
--user 1000:1000 --cap-drop ALL --security-opt no-new-privileges \
--memory 2g --cpus 2 --pids-limit 256 \
local/audit-runner:1 /bin/sh -c 'cp -a /target/. /scratch/t/ && cd /scratch/t && <command>'
The target is mounted read-only and copied into a writable scratch directory per run. The snapshot comes from git archive, so the audit sees one commit and not my working tree. Because .git is absent from the snapshot, the prompt has to state the source ref explicitly, otherwise the agent cannot say what it audited.
Gap 3: who validates the validators
The two .cjs validators are the only objective gate in the pipeline. If they run, a malformed findings.json cannot be reported as a result. If they do not, you get a report that looks structured and means nothing. So the parent runs them, not the agent, and both must exit zero before a run is done.
The same logic applies to the findings themselves. A confirmed verdict from a delegated agent is a claim. For each confirmed item I wrote a small reproduction test as a separate file against the original commit, then ran the same test against the fixed commit. Red before, green after is the only evidence I accept for a security fix.
What it costs in time
Measured on a Go codebase of moderate size:
| Profile | Wall clock | Agents |
|---|---|---|
| quick, scoped | about 40 minutes | fewer |
| quick, full | about 52 minutes | 19 |
| standard, full | about 2.9 hours | up to 60 |
The repository notes that a single run finds roughly half of what repeated runs find. My own numbers agree: the second and third runs produced findings the first did not, and one of them reversed a rejection from the first.
Things I would do differently
Do not touch git while a delegated agent is working. I ran a checkout during a run and the agent kept reading a stale HEAD, which produced findings against code that no longer existed. Fetch and reset before or after a run, never during.
Keep the agent's output directory outside the target repository. The skill defaults to a path under your home directory for a reason.
Treat the profile as a coverage decision, not a speed setting. A quick run that returns few findings may have simply not looked.
The findings from the three runs, including the one verdict that reversed between runs, are in a separate post.
If you have tried to run an agent-oriented security skill on your own projects: which requirement stopped you first, the subagents or the sandbox?
Originally published at agentjournal.dev.
Top comments (1)
Red before, green after is the rule I'd keep from this. A confirmed verdict from an agent is still a claim until a test fails on the old commit.
On the sandbox part, in smart contract repos the same risk hides in foundry.toml: with ffi = true, forge test can run any shell command, so running a client's test suite outside a sandbox is exactly the incident you describe. I'd check that flag before any agent touches the suite.
On the rejection that reversed: did the second run reach different code, or the same code and judge it differently? That tells you if you need more runs or a better verifier.