Pin the Box Before You Trust the Verdict: Environment Fingerprints for Agent Patch Gates

A green gate on an unpinned machine is a statement about that machine, not about the patch. That sounds pedantic until an agent opens eleven pull requests before lunch and every one of them is judged on a box that was created by a different script, at a different kernel version, with a different lockfile resolution. Then the gate is not measuring patch quality. It is measuring ambient drift, and nobody can tell which.

This post is about the smallest change that removes that ambiguity: make the gate box emit a fingerprint, and make the gate refuse to return a verdict when the fingerprint does not match the one you pinned. Everything else — property checks, fixture locks, flaky-test freezes — hangs off that refusal.

Why agent patches make this urgent

Human pull requests arrive slowly enough that a drifting runner usually gets noticed. Agent patches do not. They arrive in batches, they are cheap to regenerate, and the natural response to a red result is to rerun on a fresh box. That is exactly the loop that turns environment variance into false signal: the patch that passed on attempt three may have passed because a warm cache, a different CPU count, or a locale changed the test ordering.

There is a second reason. If the patch author is a model, the model is not the thing you want to audit at merge time. The gate is. So the gate has to be boring, deterministic, and explicit about when it does not know.

Step 1 — Decide what belongs in the fingerprint

A fingerprint is not a full environment capture. It is the set of fields that, if changed, could plausibly flip a test result. Choose fields you can compute cheaply and diff loudly.

Field	Why it can flip a verdict	Cost to compute
Container image digest (or OS + kernel)	libc, timer resolution, thread scheduling	one command
Language runtime + package manager version	resolution order, default warning-as-error behavior	one command
Dependency lock digest (resolved tree, not just the lockfile)	a lockfile can resolve differently across indexes	seconds
Test fixture tree digest	golden files silently regenerated by a previous run	seconds
Runner config digest (pytest.ini, conftest.py, etc.)	collection order, marker filters	milliseconds
Env allowlist digest (LANG, TZ, PYTHONHASHSEED, PATH)	hash seeding and locale change iteration and sorting	milliseconds
Resource envelope (CPU count, memory ceiling)	concurrency and timeout tests	one command

Leave out anything that changes on every run — timestamps, hostnames, absolute paths. A fingerprint that always differs is worse than none, because people learn to ignore it.

Step 2 — Compute it and pin it

The snippet below is illustrative and has not been executed for this article. It uses only the standard library so you can read it without installing anything.

# gate/fingerprint.py — illustrative, unexecuted
import hashlib, json, os, platform, shutil, subprocess

ENV_ALLOWLIST = ("PATH", "LANG", "LC_ALL", "TZ", "PYTHONHASHSEED")

def _d(b: bytes) -> str:
    return hashlib.sha256(b).hexdigest()[:16]

def file_digest(path: str) -> str:
    with open(path, "rb") as f:
        return _d(f.read())

def tree_digest(root: str) -> str:
    h = hashlib.sha256()
    for dirpath, dirnames, filenames in os.walk(root):
        dirnames.sort()
        for name in sorted(filenames):
            p = os.path.join(dirpath, name)
            h.update(os.path.relpath(p, root).encode() + b"\0")
            with open(p, "rb") as f:
                h.update(_d(f.read()).encode())
    return h.hexdigest()[:16]

def env_digest() -> str:
    h = hashlib.sha256()
    for k in ENV_ALLOWLIST:
        h.update(f"{k}={os.environ.get(k, '<unset>')}\0".encode())
    return h.hexdigest()[:16]

def build(root: str = ".") -> dict:
    return {
        "python": platform.python_version(),
        "machine": platform.machine(),
        "cpu_count": str(os.cpu_count()),
        "env": env_digest(),
        "lock": file_digest(os.path.join(root, "requirements.lock")),
        "runner_cfg": file_digest(os.path.join(root, "pytest.ini")),
        "fixtures": tree_digest(os.path.join(root, "tests", "fixtures")),
    }

if __name__ == "__main__":
    print(json.dumps(build(), indent=2, sort_keys=True))

Pin the output as gate/fingerprint.json in the repository, generated on the authoritative runner. Now the pinned file is reviewable: a diff in it is a deliberate environment change, not a mystery.

One honest caveat: the fingerprint cannot see everything. Floating-point libraries, GPU drivers, and network latency are outside it. Say so in the gate output rather than pretending coverage.

Step 3 — Make the verdict three-state

Boolean gates force a lie. When the box drifted, the truthful answer is neither pass nor fail.

# gate/verdict.py — illustrative, unexecuted
FROZEN_COVER = False  # set by the freeze ledger, see Step 5

def verdict(observed: dict, pinned: dict, exit_code: int) -> dict:
    drift = sorted(k for k in pinned if observed.get(k) != pinned[k])
    if drift:
        return {"state": "NO_VERDICT", "reason": "environment_drift", "fields": drift}
    if FROZEN_COVER and exit_code == 0:
        return {"state": "NO_VERDICT", "reason": "green_rests_on_frozen_tests"}
    if exit_code == 0:
        return {"state": "PASS"}
    return {"state": "FAIL", "reason": "test_failure"}

Wire NO_VERDICT to a different queue than FAIL. A drifted box needs a human or a rebuild, not a patch author chasing a phantom regression. If your merge bot treats NO_VERDICT as failure, you have reinvented the problem with more steps.

Step 4 — Run cheap property checks before the suite

Order the gate by cost. Property checks are assertions about the patch itself that need no test execution and catch whole classes of nonsense in milliseconds:

1. The patch touches at most N files, or declares why it exceeds N.
2. The patch does not modify gate/fingerprint.json, the fixture tree, or the runner config without an explicit env_change: true marker in the description.
3. The declared public API signature set is unchanged when the patch claims to be a fix.
4. The patch adds no network calls under tests/unit/.

These are cheap, and cheap checks that fire early keep expensive flaky suites out of the loop. They are also deterministic, which the fingerprint then certifies.

Step 5 — Lock fixtures, then freeze flakes with an expiry

A fixture that a patch can rewrite is not a fixture. Hash the fixture tree (already in the fingerprint) and fail closed: if the digest moved between the pre-patch and post-patch run, the gate returns NO_VERDICT until someone signs off on the new golden files.

Flaky tests need the same treatment, with a clock attached. A freeze entry should carry the test id, the baseline evidence (how many failures on an unmodified checkout), an owner, and an expiry date. Two rules make the ledger useful instead of decorative:

• Expired entries do not silently fall back to "run anyway." They escalate to NO_VERDICT on the next gate run.
• A frozen test cannot contribute green. If the patch modifies lines covered only by frozen tests, the overall result is NO_VERDICT, because the green is borrowed.

That second rule is the one people skip, and it is the one that prevents a freeze list from quietly becoming a coverage hole with a ticket attached.

Step 6 — Split the work across two kinds of box

The pragmatic pattern is a split: the authoritative verdict runs on a pinned, versioned runner, while exploration runs on disposable capacity you can throw away without ceremony.

Workload	Where it runs	Blocking on merge
Fingerprint computation and property checks	anywhere, cheap	only the pinned run counts
Suite sanity during patch authoring	disposable box	no
Authoritative suite run	pinned runner matched to fingerprint.json	yes
Flake reproduction attempts	disposable box, results recorded as evidence	no

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator of this account describes MonkeyCode as an open-source project offering free model access and a free server option, with a stated token allowance on the free tier. Treat all of that as time-sensitive: quotas, model availability, and server terms change, so confirm the current numbers on the project itself before you design a workflow that depends on them. The honest fit for this article is the right-hand column above — the disposable side. Free model access lowers the cost of generating candidate patches and re-drafting after a NO_VERDICT; a free server is a plausible home for the throwaway box that produces evidence without producing verdicts. Neither belongs on the authoritative path, because a pinned fingerprint is worth more than free capacity.

Limitations and who should skip this

• It does not make a flaky suite deterministic. It makes the flakiness attributable. If you want determinism, that is a different project.
• Fingerprints rot. Pin them for a release cycle, not a year, or every gate run returns drift and the team stops reading the output.
• Preemptible capacity is a bad host for long suites. If the box can vanish mid-run, the gate needs resumability or it needs to not run there.
• Skip this if you have one stable runner and a fast suite. You are already pinned by accident; write the fingerprint down and move on.
• Skip this if your merge latency budget is tighter than your suite duration. Ordering by cost helps, but a three-state gate adds a rebuild step on drift.

Check your own gate in five minutes

Run the fingerprint script twice on the box that currently issues verdicts, with the full suite in between. If the two outputs differ, your existing green results were partly reporting on the box. Fix that before you add another test-selection heuristic — a selection policy computed on an unpinned machine is just a more sophisticated way to be wrong.

If you try this with a disposable runner on the free side of the split, keep fingerprint.json generated on the pinned side and diff it in review. The day someone changes the pinned file is the day you want a second pair of eyes, not a passing build.