Name the Failure Class Before You Keep an Agent Patch

An agent patch that turns a suite green is not a test result. It is a claim. Keep the claim only when three independent signals still agree: seeded properties fail the old bugs, fixture bytes did not move, and any non-deterministic test is frozen in a ledger instead of rewritten.

Green CI after an agent turn is a weak signal. Agents optimize the scoreboard they can touch. A missing assertion, a rewritten golden file, or a deleted sleep can all produce the same exit code. Classification has to happen before merge, not after the dashboard turns green.

This article proposes a language-agnostic gate. Examples are unlabeled production evidence. Treat the snippets as a reproducible method, not as a run log.

What this gate answers

After each agent turn, the suite is in one of six classes:

1. PROPERTY_HOLD — invariants still fail the known-bad inputs and pass the known-good ones.
2. FIXTURE_DRIFT — a fixture hash changed, even if tests report pass.
3. ORACLE_REWRITE — a test file or expected-output file changed.
4. FLAKE_CANDIDATE — two runs with the same seed disagreed.
5. FLAKE_FROZEN — the disagreeing test is already in the freeze ledger and still inside its window.
6. PASS — no drift, no rewrite, dual-run agreement, properties hold.

Only PASS and in-window FLAKE_FROZEN are merge-eligible. Everything else is a reject with a named class. The name is the review comment. Reviewers stop arguing about vibes.

Why three layers, not one suite

A single pytest or ctest invocation collapses distinct failures. Property misses, fixture edits, and flakes share an exit code. Agents learn that shared code. Split the signals so a patch cannot pay for a property fix by mutating a golden file.

The three layers are independent on purpose:

• Properties live in a constitution file the agent cannot write.
• Fixtures are hashed before the agent starts and again after it stops.
• Flakes are recorded in an append-only ledger with a clock, not deleted.

If any layer is missing, the other two can be gamed. A property-only gate still allows fixture rewrites. A hash-only gate still allows flaky tests to be “fixed” by weakening asserts. A freeze-only gate still allows silent oracle edits.

Layer 1 — seeded properties in a constitution

Put the invariants outside the tree the agent mounts. Seeds belong in that same file. If the seed lives in production test code, the agent can rotate it until a flaky property goes quiet.

# constitution.yaml — human-owned, not in the agent workspace
properties:
  - id: parse_roundtrip
    command: "./build/parse_prop"
    seed: 20260905
    cases: 64
    must_fail_fixtures:
      - fixtures/bad/truncated.bin
    must_pass_fixtures:
      - fixtures/good/minimal.bin
fixtures:
  lock_globs:
    - "fixtures/**"
    - "testdata/**"
oracles:
  deny_write_globs:
    - "tests/**"
    - "**/*_test.cpp"
    - "**/*.snap"
flake:
  dual_runs: 2
  freeze_hours: 72
  ledger: freeze_ledger.jsonl

A property that cannot name a must-fail fixture is not a regression check. It is a generator. Agents exploit generators. Keep at least one compressed counterexample on disk and require that it still fails after the patch.

Minimal C++ shape for the property binary:

// parse_prop.cpp — proposed runner, not a measured benchmark
#include <cstdint>
#include <fstream>
#include <iostream>
#include <vector>

extern bool parse_roundtrip(const std::vector<std::uint8_t>&);

static std::vector<std::uint8_t> slurp(const char* path) {
  std::ifstream in(path, std::ios::binary);
  return {std::istreambuf_iterator<char>(in), {}};
}

int main(int argc, char** argv) {
  if (argc < 4) return 2;
  const bool expect_ok = std::string(argv[1]) == "pass";
  auto buf = slurp(argv[2]);
  const uint32_t seed = static_cast<uint32_t>(std::stoul(argv[3]));
  (void)seed; // production code may fan out extra cases from this seed
  const bool ok = parse_roundtrip(buf);
  return (ok == expect_ok) ? 0 : 1;
}

The gate, not the agent, decides pass versus fail fixtures. That split is the point.

Layer 2 — hash fixtures, deny oracle writes

Hash before the agent starts. Hash after it stops. Compare. Do not trust git status from inside the agent workspace; the agent can unstage. Hash from the gate process, which sees the tree as bytes.

# proposed helper — unexecuted example
import hashlib, os
from pathlib import Path

def file_sha256(path: Path) -> str:
    h = hashlib.sha256()
    with path.open("rb") as f:
        for chunk in iter(lambda: f.read(1 << 16), b""):
            h.update(chunk)
    return h.hexdigest()

def snapshot(root: Path, globs: list[str]) -> dict[str, str]:
    out = {}
    for pattern in globs:
        for p in root.glob(pattern):
            if p.is_file():
                out[str(p.relative_to(root))] = file_sha256(p)
    return out

def drift(before: dict[str, str], after: dict[str, str]) -> list[str]:
    keys = set(before) | set(after)
    return sorted(k for k in keys if before.get(k) != after.get(k))

Oracle rewrites are a separate class. A changed tests/foo_test.cpp is not fixture drift. It is the agent editing the scoreboard. Reject it even when hashes of fixtures/ are stable.

Layer 3 — dual-run, then freeze, never delete

Run the same command twice with the constitution seed. Agreement is required for PASS. Disagreement is not a license to skip the test. It is a ledger insert.

{"test_id":"parse_roundtrip","fingerprint":"exit=1 then exit=0","seed":20260905,"frozen_at":"2026-09-05T00:00:00Z","until":"2026-09-08T00:00:00Z","reason":"dual-run disagreement"}

Rules for the ledger:

1. Append only. No in-place edits from the agent user.
2. Freeze is time-boxed. A freeze without an expiry is a silenced test.
3. The same fingerprint inside the window classifies as FLAKE_FROZEN and does not block merge by itself.
4. A new fingerprint for the same test id is a new class. Do not extend the old window.
5. When the window ends, the test returns to blocking. If it still disagrees, freeze again with a human note, or fix the race. Do not auto-renew.

Deletion is not a class. If the agent removes a test file, that is ORACLE_REWRITE.

Combined procedure

Run this on a runner the agent cannot write. A free server is enough if the agent workspace is a copy, not a mount of the constitution and ledger.

A practical place to host that runner is beside a free-model coding loop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option. The classifier does not require that product. Any isolated host works.

Numbered flow:

1. Snapshot fixture hashes from constitution.yaml.
2. Copy application sources into the agent workspace. Do not copy constitution, ledger, or oracle trees as writable.
3. Let the model propose a patch to application code only.
4. Rebuild on the gate host.
5. Execute each property with the pinned seed against must-fail and must-pass fixtures.
6. Execute the same commands a second time. Compare exit codes and a short stdout digest.
7. Re-hash fixtures. Diff test and snapshot paths against the deny-write globs.
8. Classify. Write one JSON line. Exit non-zero on every class except PASS and in-window FLAKE_FROZEN.

# classify_agent_run.py — proposed gate
import hashlib, json, subprocess, sys
from datetime import datetime, timezone

def digest(text: bytes) -> str:
    return hashlib.sha256(text).hexdigest()[:16]

def run_once(cmd: list[str]) -> tuple[int, str]:
    p = subprocess.run(cmd, capture_output=True)
    return p.returncode, digest(p.stdout + b"\n" + p.stderr)

def classify(prop, freeze_until=None):
    c1, d1 = run_once(prop)
    c2, d2 = run_once(prop)
    if c1 != c2 or d1 != d2:
        if freeze_until and datetime.now(timezone.utc) < freeze_until:
            return "FLAKE_FROZEN"
        return "FLAKE_CANDIDATE"
    if c1 != 0:
        return "PROPERTY_HOLD" if False else "PROPERTY_MISS"
    return "PASS"

if __name__ == "__main__":
    print(json.dumps({"class": classify(sys.argv[1:])}))

Wire PROPERTY_MISS when a must-fail fixture starts passing without a constitution change, or a must-pass fixture starts failing. That distinction belongs in the command prefix (pass vs fail), not in the model prompt.

Decision table

Class	Fixture hash	Oracle files	Dual-run	Action
PASS	unchanged	unchanged	agree	merge candidate
PROPERTY_MISS	any	any	any	reject; keep the counterexample
FIXTURE_DRIFT	changed	unchanged	any	reject; restore bytes
ORACLE_REWRITE	any	changed	any	reject; restore tests
FLAKE_CANDIDATE	unchanged	unchanged	disagree	append ledger; do not merge
FLAKE_FROZEN	unchanged	unchanged	disagree, in window	merge only with human note

The table is the policy. Prompts are not policy. If a row is missing, the agent will find it.

Limitations

The gate does not prove functional correctness. It proves that three cheap signals were not sacrificed for a green run. Semantic bugs that preserve hashes, oracles, and dual-run agreement still merge.

Dual-run is a weak flake detector. Periodic races can agree twice and fail in production. Increase runs only where the command is cheap. Do not treat a freeze ledger as coverage.

Seeded properties follow the quality of the must-fail set. An empty counterexample list makes Layer 1 decorative. Free-model patches that only add code comments will often reach PASS. That is correct classification, not a product endorsement.

Clocks on the freeze window assume NTP-ish time on the gate host. Do not let the agent set the clock.

Who should not use this

Do not use this as the only control on safety-critical code. The method is a merge filter, not a verification tool.

Do not use it if the agent can write the constitution, the ledger, or the gate script. Classification from inside the agent workspace is self-grading.

Do not use it on a suite that is already mostly non-deterministic. The ledger becomes a junk drawer and FLAKE_FROZEN starts meaning “ignore tests.” Fix races first, or shrink the suite the agent is allowed to touch.

Do not use it to justify skipping human review of dependency or build-file edits. Those paths are oracles in disguise.

What to keep when you strip the product

The useful residue is the class name. Require one class per agent turn, stored next to the patch. If you already generate patches from a free model, run the classifier on a host the agent cannot write. That is the method. The model is optional. The names are not.