Put the Verifier Where the Agent Can't Write: A Remote Testing Packet for Agent Patches

Start with the boundary, not the tests

Most agent-patch testing advice argues about which tests to run. The bigger variable is where they run. If the agent that produced the patch can also write to tests/, conftest.py, or the runner's environment, then a green result is a claim about the agent's willingness to leave the suite alone — not a claim about the code.

I have spent the last several months building harnesses for this problem: permutation sweeps, seed sweeps, portability ledgers, a shuffle-and-bisect runner, a falsifiability gate. Each one removed a specific way a green run could lie. This article covers the one that sat underneath all of them and that I kept postponing: the trust boundary between the patch author and the patch verifier.

The rule I now apply is short. The verifier runs on a machine the agent cannot write to, and the only thing that crosses back is a bounded failure digest. Everything below is the mechanics of that rule.

1. Three invariants of a verifier the agent cannot touch

Before any code, agree on what the remote side is allowed to assume. Three invariants do most of the work:

1. Test-tree hash pin. The verifier recomputes a hash over the test tree and compares it to a manifest built before the agent ran. Mismatch is a hard reject, not a warning.
2. No install at run time. Dependencies come from a lockfile with hashes, resolved before the patch is applied. Otherwise the agent's patch can add a package that quietly changes collection order or skips tests.
3. Environment fingerprint. Python version, platform, and the sorted list of installed distributions get recorded with the verdict. A green with an unrecorded environment is not evidence you can move to another machine.

Invariant 2 is the one people skip. It is also the one that lets a patch pass locally and fail in CI forever.

2. Build the packet before the agent runs

The manifest has to be created from a state you control. Run this on the base commit, not on the agent's branch:

# tools/packet.py — build a manifest over the test tree
import hashlib, json, pathlib, sys

def tree_hash(root: pathlib.Path) -> str:
    h = hashlib.sha256()
    for p in sorted(root.rglob("*.py")):
        if "__pycache__" in p.parts:
            continue
        h.update(p.relative_to(root).as_posix().encode())
        h.update(p.read_bytes())
    return h.hexdigest()

def main() -> int:
    tests, lock, out = (pathlib.Path(a) for a in sys.argv[1:4])
    manifest = {
        "tests_hash": tree_hash(tests),
        "lock_hash": hashlib.sha256(lock.read_bytes()).hexdigest(),
        "python": sys.version.split()[0],
    }
    out.parent.mkdir(parents=True, exist_ok=True)
    out.write_text(json.dumps(manifest, indent=2, sort_keys=True))
    print(out)
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

python tools/packet.py tests requirements.lock .verify/manifest.json

Two details that matter: paths are sorted and normalized to POSIX form, so a Windows checkout and a Linux runner agree; and __pycache__ is excluded, because bytecode churn is not a test change. If your suite has fixtures outside tests/, widen the root or you have pinned nothing.

3. Verify on the remote side and refuse to proceed on mismatch

The verifier is deliberately dumb. It checks the hash, then runs pytest, then emits one JSON object. It has no repair logic, because repair logic on the verifier is just the agent again with extra steps.

# verify.py — runs on the clean host
import json, pathlib, subprocess, sys

from tools.packet import tree_hash

MANIFEST = pathlib.Path(".verify/manifest.json")
FROZEN = pathlib.Path(".verify/frozen.json")

def verdict(ok, reason, extra=None):
    payload = {"verdict": "pass" if ok else "reject", "reason": reason}
    payload.update(extra or {})
    print(json.dumps(payload))
    raise SystemExit(0 if ok else 1)

def main() -> None:
    m = json.loads(MANIFEST.read_text())
    if tree_hash(pathlib.Path("tests")) != m["tests_hash"]:
        verdict(False, "TESTS_MUTATED")

    ids = json.loads(FROZEN.read_text())["ids"] if FROZEN.exists() else []
    deselect = [arg for i in ids for arg in ("--deselect", i)]

    proc = subprocess.run(
        [sys.executable, "-m", "pytest", "-q", "--tb=line", "-p", "no:cacheprovider", *deselect],
        capture_output=True, text=True, timeout=900,
    )
    tail = proc.stdout.strip().splitlines()[-8:]
    verdict(proc.returncode == 0, "PYTEST", {
        "frozen": len(ids),
        "tail": tail,
        "exit": proc.returncode,
    })

if __name__ == "__main__":
    main()

Note -p no:cacheprovider. The cache directory is a write channel: an agent-edited lastfailed file changes what the next run considers worth running. Disable it rather than trust it.

4. Return a digest, not a log

The feedback channel is where token budgets quietly leak. If you paste a 3,000-line pytest log back into the model, you are paying for the agent to read noise, and you are inviting it to fix symptoms. Cap the return payload at something small and structured:

{"verdict":"reject","reason":"PYTEST","exit":1,
 "frozen":2,"tail":["tests/test_queue.py:41: AssertionError: expected 3 got 2"]}

This is the part that interacts with free model access. A bounded digest makes a cheap model usable for the repair loop, because every iteration has a fixed and small input cost. MonkeyCode's free model access is where I run this loop, and its free server option is where I run verify.py — a disposable host that the agent has no write path to is exactly the shape this workflow needs. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free tier is currently advertised at 10M tokens plus a free server option. Treat both as time-sensitive terms: they are availability claims, not stable architecture, and you should confirm the current numbers on the project page before you size a budget around them. The workflow below is designed so that a change in quota does not break it.

5. Give each patch an evidence budget

Once verification costs something, cost becomes a design constraint you can measure. Log every run:

run_id,patch_id,subset,tokens_in,tokens_out,wall_s,verdict
r-0007,p-31,queue tests,4120,380,19,reject
r-0008,p-31,queue tests,4301,512,18,reject
r-0009,p-31,full,8804,240,74,pass

Two rules fall out of that ledger, and they are policy, not measurement:

• Six-run cap. I allow at most six verifier runs per patch. Past that, the patch is too large to verify, and the fix is to split the patch — not to raise the cap. In my own use, the failures that survive five repair attempts are almost always mis-scoped rather than mistyped.
• Full-suite escalation is earned. The agent may request full only after a subset run has produced a specific failing node id. "Run everything and tell me what's wrong" is how a token budget disappears in one turn.

6. Freeze flaky tests by evidence, not by calendar

A flake freeze without a resurrection condition is a slow deletion. Every entry needs four fields:

{"ids":["tests/test_http.py::test_retry_after"],
 "entries":[{"id":"tests/test_http.py::test_retry_after","first_seen":"r-0007",
             "owner":"http-layer","unfreeze_cmd":"pytest tests/test_http.py::test_retry_after -x -q"}]}

• Unfreeze requires evidence, not a date. The entry leaves the frozen set after K consecutive green runs of unfreeze_cmd on the verifier host. I use K = 10. A date-based expiry just moves the flake back into the suite on a day nobody is watching.
• Frozen-set growth is a signal. If the set grows by more than two entries per week, the suite is being hollowed out and the pass rate is no longer comparable across weeks.
• Reject on deselect failure. If pytest errors because a frozen node id no longer exists, that is a reject. Stale freeze entries hide renames, and a rename is often how a real test disappears.

Decision table: what runs where

Evidence needed	Agent's machine	Remote verifier	Why
Fast unit loop on changed module	yes	must re-run	local runs guide the agent, they do not count
Test-tree / lockfile integrity	never trusted	only here	this is the whole point of the boundary
Flake detection (repeat runs)	no	yes, capped	expensive, and needs the clean environment
Cross-machine portability	no	yes	fingerprint makes the record portable
Integration, perf, load	no	out of scope	needs services and time; hash pinning does not help

Limitations and who should not use this

• It does not measure assertion strength. A hash pin proves the tests were not edited; it proves nothing about whether they assert the right thing. That is a separate gate, and I have written about classifying tautological tests for exactly this reason.
• Single-host noise is not excluded. The verifier filters edits, not hardware variance. If your failure mode is timing or memory pressure, you need repeated runs on more than one host before you call anything green.
• Skip it if you have one service, one developer, and no agent in the loop. The overhead of a packet, a manifest, and a ledger is not worth it against a suite you can read in full.
• Skip it if you cannot pin dependencies. Without lockfile hashes, invariant 2 is decoration, and you should fix packaging before you build a verifier.

If you want to test whether the boundary actually changes your results without provisioning anything, the project's free server option is the cheapest place I know of to run one honest verifier and see what your suite reports when the agent is not standing next to it.

Further reading: pip hash-checking mode at https://pip.pypa.io/en/stable/cli/pip_install/#hash-checking-mode and pytest's cache documentation at https://docs.pytest.org/en/stable/how-to/cache.html, which is the write channel disabled in step 3.