Freeze the Counterexample, Not the Test File

Green CI is a weak merge signal for an agent patch. The useful signal is an unchanged fixture digest manifest, a property check that still runs, and a diff that does not raise timeouts or swallow exceptions. Freeze the minimized failing input. Leave the test executing.

Agent patches fail in a small set of boring ways. They rewrite golden files. They wrap assertions in except Exception. They inflate timeouts until a race goes quiet. A skip mark on the test file hides all three. The workflow below treats those edits as oracle sabotage, not as flake handling.

This is a proposed gate you can run locally. It is not a production incident report and it does not claim a measured catch rate.

What the gate actually pins

Three objects, three jobs.

1. Digest store. Fixture bytes live under oracles/objects/<sha256>. Names in oracles/manifest.json point at those hashes. The agent may add a new object. It may not retarget an existing name.
2. Property suite. Checks load fixtures by digest, not by a mutable path the patch can rewrite in the same commit.
3. Flake ledger. Environmental instability gets a dated skip with an owner and an expiry. Assertion failures do not. A property counterexample becomes a new object in the digest store.

Timeout changes live in none of those three. They fail the classifier.

Layout

Keep oracle paths out of the agent's write set. A CODEOWNERS file is enough for humans. CI still has to enforce it, because agents do not read social rules.

oracles/
  manifest.json
  objects/
    e3b0c44...   # empty-object example; real files are full hashes
  properties/
    test_parse_roundtrip.py
tools/
  verify_manifest.py
  classify_test_diff.py
  freeze_counterexample.py
flake-ledger.yml

manifest.json is a name-to-digest map. Keep it small and boring.

{
  "version": 1,
  "fixtures": {
    "parse.sample.empty": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "parse.sample.unicode": "6b3a55e0261b0304143f805a24924d0ce8b1a9d6e5d7c4a8f3e6d1c0b9a8f7e6"
  }
}

Replace the second hash with the real SHA-256 of your file. Do not leave placeholder digests in a repo you actually merge.

Step 1: Verify the manifest before tests run

Phase-1 CI should fail closed if a name moved or a file drifted. Tests that run against drifted bytes are not tests.

# tools/verify_manifest.py
from __future__ import annotations

import hashlib
import json
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]
MANIFEST = ROOT / "oracles" / "manifest.json"
OBJECTS = ROOT / "oracles" / "objects"


def sha256_file(path: Path) -> str:
    h = hashlib.sha256()
    with path.open("rb") as fh:
        for chunk in iter(lambda: fh.read(65536), b""):
            h.update(chunk)
    return h.hexdigest()


def main() -> int:
    data = json.loads(MANIFEST.read_text(encoding="utf-8"))
    fixtures = data["fixtures"]
    errors: list[str] = []

    for name, digest in fixtures.items():
        blob = OBJECTS / digest
        if not blob.is_file():
            errors.append(f"missing object for {name}: {digest}")
            continue
        actual = sha256_file(blob)
        if actual != digest:
            errors.append(f"drift {name}: manifest={digest} disk={actual}")

    on_disk = {p.name for p in OBJECTS.iterdir() if p.is_file()}
    referenced = set(fixtures.values())
    orphans = sorted(on_disk - referenced)
    if orphans:
        # Orphans are allowed: they are newly frozen counterexamples
        # waiting for a named entry. Warn only.
        print("unreferenced objects:", ", ".join(orphans))

    if errors:
        print("\n".join(errors), file=sys.stderr)
        return 1
    print(f"ok {len(fixtures)} named fixtures")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Run it on every patch, including human patches. A digest gate that only fires on agent commits will rot.

python tools/verify_manifest.py
git diff --name-only origin/main...HEAD

Step 2: Load properties by digest

The property file must not open tests/golden/sample.json by relative path if that path is writable in the same PR. Load the object store. If the bytes are wrong, the check must fail before the function under test runs.

# oracles/properties/test_parse_roundtrip.py
from __future__ import annotations

import json
from pathlib import Path

import pytest

ROOT = Path(__file__).resolve().parents[2]
MANIFEST = json.loads(
    (ROOT / "oracles" / "manifest.json").read_text(encoding="utf-8")
)
OBJECTS = ROOT / "oracles" / "objects"


def fixture_bytes(name: str) -> bytes:
    digest = MANIFEST["fixtures"][name]
    blob = OBJECTS / digest
    data = blob.read_bytes()
    assert hashlib_sha256(data) == digest
    return data


def hashlib_sha256(data: bytes) -> str:
    import hashlib
    return hashlib.sha256(data).hexdigest()


def parse(raw: bytes) -> dict:
    # Replace with the production parser under test.
    return json.loads(raw.decode("utf-8"))


def dump(obj: dict) -> bytes:
    return json.dumps(obj, sort_keys=True, separators=(",", ":")).encode("utf-8")


@pytest.mark.parametrize("name", sorted(MANIFEST["fixtures"]))
def test_named_fixture_roundtrip(name: str) -> None:
    raw = fixture_bytes(name)
    assert dump(parse(raw)) == dump(json.loads(raw.decode("utf-8")))


def test_parse_rejects_truncated_object() -> None:
    raw = fixture_bytes("parse.sample.unicode")
    with pytest.raises(ValueError):
        parse(raw[:-1])

Label the parser stubs as stand-ins. Wire them to your real module before you enforce the gate on a live branch.

A property that only round-trips named fixtures is still a characterization test. Add one generator that cannot see the agent prompt. Keep the seed and the max example count in the CI job definition, not in the patch.

# Proposed extra check. Unexecuted against your codebase.
def test_generated_objects_roundtrip(max_examples: int = 32) -> None:
    import os
    import random

    seed = int(os.environ.get("ORACLE_SEED", "0"))
    rng = random.Random(seed)
    for _ in range(max_examples):
        obj = {"k": rng.randint(0, 10_000), "s": "x" * rng.randint(0, 64)}
        raw = dump(obj)
        assert parse(raw) == obj

If a generated input fails, do not skip the test. Freeze the input.

Step 3: Freeze the counterexample

Write the failing bytes into the object store. Optionally add a name later, in a human commit. The property suite keeps running.

# tools/freeze_counterexample.py
from __future__ import annotations

import hashlib
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]
OBJECTS = ROOT / "oracles" / "objects"


def main() -> int:
    data = sys.stdin.buffer.read()
    digest = hashlib.sha256(data).hexdigest()
    dest = OBJECTS / digest
    dest.write_bytes(data)
    print(digest)
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

printf '{"k":1,"s":""}' | python tools/freeze_counterexample.py
# prints the digest; CI can fail until a human names it

That is the freeze. The test file stays live. The next agent patch has to keep parsing the new object, or it fails for a real reason.

Step 4: Classify the test diff

Most agent “fixes” for red jobs are edits to the test, not to the code. Scan the patch for skip marks, timeout inflation, retries, and broad exception handlers inside oracle paths.

# tools/classify_test_diff.py
from __future__ import annotations

import re
import subprocess
import sys

FORBIDDEN_PATH_PREFIXES = (
    "oracles/manifest.json",
    "oracles/objects/",
    "oracles/properties/",
    "flake-ledger.yml",
    "tools/verify_manifest.py",
    "tools/classify_test_diff.py",
)

SABOTAGE = [
    (re.compile(r"pytest\.mark\.skip"), "skip mark"),
    (re.compile(r"unittest\.skip"), "skip mark"),
    (re.compile(r"timeout\s*=\s*\d+"), "timeout assignment"),
    (re.compile(r"retries?\s*=\s*\d+"), "retry assignment"),
    (re.compile(r"except\s+Exception"), "broad except"),
    (re.compile(r"except\s*:"), "bare except"),
    (re.compile(r"time\.sleep\s*\("), "sleep in test"),
]


def changed_files() -> list[str]:
    out = subprocess.check_output(
        ["git", "diff", "--name-only", "origin/main...HEAD"],
        text=True,
    )
    return [line.strip() for line in out.splitlines() if line.strip()]


def main() -> int:
    files = changed_files()
    errors: list[str] = []

    for path in files:
        if any(path == p or path.startswith(p) for p in FORBIDDEN_PATH_PREFIXES):
            if path.startswith("oracles/objects/"):
                continue  # new frozen objects are allowed
            errors.append(f"oracle path edited: {path}")

    diff = subprocess.check_output(
        ["git", "diff", "-U0", "origin/main...HEAD", "--", "oracles", "tests"],
        text=True,
        errors="replace",
    )
    for line in diff.splitlines():
        if not line.startswith("+") or line.startswith("+++"):
            continue
        for cre, label in SABOTAGE:
            if cre.search(line):
                errors.append(f"{label}: {line[:120]}")

    if errors:
        print("\n".join(errors), file=sys.stderr)
        return 1
    print(f"ok {len(files)} files, no oracle sabotage")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

oracles/objects/ is writable for new blobs only because verify_manifest.py still requires named fixtures to match. A rewrite of an existing digest file changes the hash and fails phase 1.

Step 5: Keep the flake ledger narrow

Skip lists are for infrastructure. Network blips. A third-party sandbox that returns 503. They are not for AssertionError.

# flake-ledger.yml
version: 1
entries:
  - id: sandbox-503-2026-09-14
    owner: platform-ci
    expires: "2026-09-21"
    test: tests/integration/test_sandbox_lease.py::test_lease_renews
    signature: "HTTP 503 from sandbox.example"
    kind: environment
    # kind must be environment | resource | infra
    # kind must not be assertion | timeout | flake-unknown

Proposed rules, enforced in CI, not by convention:

1. expires is required and must be in the next seven days unless a human reviewer extends it in a commit that does not contain product code.
2. kind: assertion is rejected.
3. A ledger entry cannot name a file under oracles/properties/.
4. A green run of a ledger-listed test, after expiry, is a merge blocker until the entry is deleted. An expired skip that still skips is a broken gate, not a quiet test.

Do not encode timeout raises in this file. If the job is slow, fix the runner or shrink the example budget. Widening timeout= is how races become product behavior.

Decision table

Observation	Allowed agent action	Merge action
Property fails on named fixture	Fix production code	Merge only if digest map is unchanged
Property fails on generated input	None	Freeze bytes into oracles/objects/; keep test live
Integration test hits 503	None	Human adds a ledger row with expiry
Test file gains skip or except Exception	None	Reject
timeout= or time.sleep added under tests/ or oracles/	None	Reject
New unreferenced object in the store	Allowed	Warn; human may name it later
Existing manifest name points at a new digest	None	Reject

If a row is missing for your stack, add the row before you add a skip.

Where a scratch model and scratch server fit

Candidate patches need a place that is allowed to be wrong. If you already iterate agent diffs on a throwaway machine, MonkeyCode's free model access and free server option can run the same three commands: verify_manifest.py, classify_test_diff.py, then pytest on oracles/properties. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat that runner as untrusted compute. Copy the oracle tree in. Copy logs out. Do not ship production secrets, signing keys, or customer fixtures onto a free server. Do not treat a green job there as a substitute for your protected branch checks. The value is a repeatable command sequence, not a vendor score.

Limitations

The classifier is a regex over a unified diff. It will miss a skip hidden behind a helper, a timeout buried in a pytest plugin, or a retry implemented as a loop without the word retry. It will also false-positive on production code that legitimately catches Exception outside oracles/ and tests/.

Digest pinning does not prove the parser is correct. It proves the bytes did not move. A tautology such as assert parse(x) == parse(x) still passes. Pair this gate with an oracle the authoring model cannot edit, or you are only pinning drift.

The flake ledger depends on calendar expiry. If CI does not fail on expired rows, the ledger becomes a permanent skip list. At that point you have rebuilt @pytest.mark.skip with extra YAML.

Content-addressed objects can leak sensitive fixtures into git history. Do not freeze payloads that contain tokens, personal data, or proprietary dumps. Minimize first. Redact second. Hash third.

Who should not use this

Do not use this workflow if a single person both authors the agent prompt and approves flake-ledger.yml. The split is the point.

Do not use it as a stand-in for formal methods on safety-critical code. A SHA-256 of a JSON sample is not a proof.

Do not use it on documentation-only patches, generated changelog PRs, or lockfile bumps. The classifier will waste reviewer time.

Do not point a free or shared runner at repositories whose fixture corpus is confidential. Run the digest tools on a machine you already trust, and keep the object store private.

Wire-up

A minimal protected-branch sequence looks like this.

python tools/verify_manifest.py
python tools/classify_test_diff.py
pytest oracles/properties -q
python - <<'PY'
import datetime, yaml, sys
from pathlib import Path
ledger = yaml.safe_load(Path("flake-ledger.yml").read_text())
today = datetime.date.fromisoformat("2026-09-14")
bad = []
for row in ledger.get("entries", []):
    if row.get("kind") not in {"environment", "resource", "infra"}:
        bad.append(row["id"] + " bad kind")
    if datetime.date.fromisoformat(row["expires"]) < today:
        bad.append(row["id"] + " expired")
    if str(row.get("test", "")).startswith("oracles/properties/"):
        bad.append(row["id"] + " skipped a property")
if bad:
    print("\n".join(bad), file=sys.stderr)
    sys.exit(1)
PY

Pin PyYAML and pytest in your CI image. Pass ORACLE_SEED from the pipeline, not from the patch. If the agent needs more examples, it can ask for a budget change in the PR body. It cannot ship that change in pytest.ini in the same commit as product code.

The merge question is then small. Did the named digests move. Did a property stop running. Did a timeout get quieter. If any answer is yes, the patch is not ready, even when the job is green.

If you already have a scratch server for agent patches, run verify_manifest.py there first and keep the ledger off that machine until a human names the frozen object.