Pin Failure Modes Before Merging an Agent Patch

A green suite on an agent patch is a weak merge signal. It shows that assertions the model could read still hold. It does not show that failures still fail in the same way.

The useful gate is smaller. Pin the failure modes. Keep negative-path fixtures out of the agent's write set. Freeze flaky timing tests instead of deleting them. Merge only when the exception taxonomy is unchanged, or when a human rewrites that contract on purpose.

Why happy-path green misleads

Agent patches optimize for visible tests. That is the loop they are scored on. When a timeout, a 409, or a ValueError is not asserted, the model can swallow it, remap it, or wrap it in a retry. The suite stays green. Callers break.

This is not a claim about model quality. It is a claim about incentive. Tests the agent can edit are not an independent oracle. Named failure modes are.

The rest of this article is a proposed workflow. Treat the code as a starting pack, not a report of a production run.

The merge artifact: a failure-mode pack

Keep three human-owned paths next to the repo, not inside the patch branch's default write set.

1. failure_modes/catalog.json — the taxonomy.
2. failure_modes/fixtures/ — byte-stable negative inputs.
3. failure_modes/flaky_freeze.yml — tests that may be skipped, never deleted, with an expiry.

The agent may read these files. It must not write them. If your review tool cannot enforce path ownership, fail CI when those paths appear in git diff --name-only.

Catalog shape

{
  "version": 1,
  "modes": [
    {
      "id": "parse.trailing-comma",
      "fixture": "fixtures/parse_trailing_comma.json",
      "exc_type": "app.errors.ParseError",
      "error_code": "PARSE_TRAILING_COMMA",
      "http_status": 400
    },
    {
      "id": "quota.exceeded",
      "fixture": "fixtures/quota_exceeded.json",
      "exc_type": "app.errors.QuotaError",
      "error_code": "QUOTA_EXCEEDED",
      "http_status": 429
    }
  ]
}

Each row is a contract. The patch may change implementation. It may not change exc_type, error_code, or http_status unless a human updates the catalog in a separate commit.

Step 1 — Record what currently fails

Do this on main, before the agent branch exists.

mkdir -p failure_modes/fixtures tools
git checkout main
python tools/record_failure_modes.py --catalog failure_modes/catalog.json --out failure_modes/baseline.json

Proposal for the recorder:

# tools/record_failure_modes.py
# Proposal / unexecuted example.
from __future__ import annotations

import importlib
import json
from pathlib import Path


def load_exc(path: str):
    mod, name = path.rsplit(".", 1)
    return getattr(importlib.import_module(mod), name)


def invoke(fixture_path: Path):
    from app.entry import handle  # project-specific seam
    payload = json.loads(fixture_path.read_text())
    return handle(payload)


def record(catalog_path: Path) -> dict:
    catalog = json.loads(catalog_path.read_text())
    rows = []
    for mode in catalog["modes"]:
        fixture = catalog_path.parent / mode["fixture"]
        expected = load_exc(mode["exc_type"])
        try:
            result = invoke(fixture)
        except expected as exc:
            rows.append({
                "id": mode["id"],
                "status": "failed_as_specified",
                "error_code": getattr(exc, "code", None),
                "http_status": getattr(exc, "http_status", None),
            })
            continue
        except Exception as exc:
            rows.append({
                "id": mode["id"],
                "status": "wrong_exception",
                "got_type": type(exc).__qualname__,
            })
            continue
        rows.append({
            "id": mode["id"],
            "status": "unexpected_success",
            "result": str(result),
        })
    return {"version": catalog["version"], "rows": rows}


if __name__ == "__main__":
    import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument("--catalog", type=Path, required=True)
    parser.add_argument("--out", type=Path, required=True)
    args = parser.parse_args()
    args.out.write_text(json.dumps(record(args.catalog), indent=2) + "\n")

The output is the baseline. Commit failure_modes/baseline.json on main. That file is human-owned too.

Step 2 — Property-check the taxonomy, not the message text

Message strings drift. Types and codes should not. A property here is a quantified assertion over the catalog, not a single example.

# tests/test_failure_mode_properties.py
# Proposal / unexecuted example.
import json
from pathlib import Path

import pytest

from tools.record_failure_modes import invoke, load_exc

CATALOG = json.loads(Path("failure_modes/catalog.json").read_text())


@pytest.mark.parametrize("mode", CATALOG["modes"], ids=lambda m: m["id"])
def test_failure_mode_still_fails_with_pinned_type(mode):
    fixture = Path("failure_modes") / mode["fixture"]
    expected = load_exc(mode["exc_type"])
    with pytest.raises(expected) as captured:
        invoke(fixture)
    exc = captured.value
    if mode.get("error_code") is not None:
        assert getattr(exc, "code") == mode["error_code"]
    if mode.get("http_status") is not None:
        assert getattr(exc, "http_status") == mode["http_status"]

This is the opposite of a tautology. The fixture is a real rejected payload. The assertion names a type the implementation must raise. If the agent rewrites handle() to return {"ok": false}, the property fails.

Do not assert on str(exc). Agents "fix" flaky wording by editing copy. That is not a failure-mode pin.

Step 3 — Lock fixtures by digest

A fixture the agent can rewrite is an oracle the agent owns. Hash the files. Compare on CI.

# tools/fixture_lock.py
# Proposal / unexecuted example.
import hashlib
import json
from pathlib import Path

ROOT = Path("failure_modes/fixtures")
LOCK = Path("failure_modes/fixtures.lock.json")


def digest_tree(root: Path) -> dict:
    out = {}
    for path in sorted(root.rglob("*")):
        if path.is_file():
            rel = str(path.relative_to(root))
            out[rel] = hashlib.sha256(path.read_bytes()).hexdigest()
    return out


def main() -> int:
    current = digest_tree(ROOT)
    if not LOCK.exists():
        LOCK.write_text(json.dumps(current, indent=2) + "\n")
        print("wrote new lock")
        return 0
    expected = json.loads(LOCK.read_text())
    if current != expected:
        print("fixture lock mismatch")
        print("expected", json.dumps(expected, indent=2))
        print("current", json.dumps(current, indent=2))
        return 1
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

CI rule: python tools/fixture_lock.py must exit 0 on agent PRs. A catalog change and a lock change land together, in a human commit, with a one-line reason.

Step 4 — Freeze flaky tests. Do not delete them

Agent patches often "stabilize" CI by removing test_upstream_deadline or by widening a sleep. That shrinks the suite. It does not fix the race.

Keep a freeze file:

# failure_modes/flaky_freeze.yml
# Proposal / policy example, not a measured flake rate.
rules:
  - nodeid: "tests/test_timeout.py::test_upstream_deadline"
    reason: "timing depends on shared runner load"
    expires: "2026-10-10"
    allow_delete: false
  - nodeid: "tests/test_retry.py::test_eventual_connect"
    reason: "dns lookup on the runner is not the unit"
    expires: "2026-10-10"
    allow_delete: false

Enforcement:

# tools/check_flaky_freeze.py
# Proposal / unexecuted example.
from __future__ import annotations

import datetime as dt
import subprocess
import sys
from pathlib import Path

import yaml

FREEZE = Path("failure_modes/flaky_freeze.yml")


def deleted_test_names() -> set[str]:
    raw = subprocess.check_output(
        ["git", "diff", "-U0", "origin/main...HEAD", "--", "tests"],
        text=True,
    )
    removed = set()
    for line in raw.splitlines():
        if line.startswith("-def test_"):
            removed.add(line[len("-def "):].split("(")[0])
    return removed


def main() -> int:
    doc = yaml.safe_load(FREEZE.read_text())
    today = dt.date.today()
    frozen_names = {rule["nodeid"].split("::")[-1] for rule in doc["rules"]}
    blocked = sorted(deleted_test_names() & frozen_names)
    if blocked:
        print("frozen tests deleted in this patch:", blocked)
        return 1
    expired = []
    for rule in doc["rules"]:
        expires = dt.date.fromisoformat(rule["expires"])
        if expires < today:
            expired.append(rule["nodeid"])
    if expired:
        print("freeze expired; re-triage before merge:", expired)
        return 1
    return 0


if __name__ == "__main__":
    sys.exit(main())

The freeze is a lease. When it expires, a human either deletes the test with a reason, rewrites it as a property, or extends the lease. The agent does not get a vote.

Step 5 — Diff the taxonomy on the patch branch

git checkout agent-patch
python tools/record_failure_modes.py --catalog failure_modes/catalog.json --out /tmp/head.json
python tools/diff_failure_modes.py --base failure_modes/baseline.json --head /tmp/head.json

# tools/diff_failure_modes.py
# Proposal / unexecuted example.
import json
import sys
from pathlib import Path


def index(doc):
    return {row["id"]: row for row in doc["rows"]}


def main(base_path: Path, head_path: Path) -> int:
    base = index(json.loads(base_path.read_text()))
    head = index(json.loads(head_path.read_text()))
    rc = 0
    for mode_id, brow in base.items():
        hrow = head.get(mode_id)
        if hrow is None:
            print(f"MISSING {mode_id}")
            rc = 1
            continue
        if brow != hrow:
            print(f"CHANGED {mode_id}")
            print("  base", brow)
            print("  head", hrow)
            rc = 1
    extra = sorted(set(head) - set(base))
    if extra:
        print("EXTRA_IDS", extra)
        rc = 1
    return rc


if __name__ == "__main__":
    args = sys.argv
    raise SystemExit(main(Path(args[args.index("--base") + 1]), Path(args[args.index("--head") + 1])))

Decision rule: any CHANGED, MISSING, or EXTRA_IDS blocks merge. Extra IDs look like progress. They are not. They mean the recorder observed a mode the catalog does not own yet. Add it in a human commit, or drop it.

A single make target keeps the gate boring:

# Proposal
.PHONY: failure-modes
failure-modes:
    python tools/fixture_lock.py
    python tools/check_flaky_freeze.py
    pytest -q tests/test_failure_mode_properties.py
    python tools/record_failure_modes.py --catalog failure_modes/catalog.json --out /tmp/head.json
    python tools/diff_failure_modes.py --base failure_modes/baseline.json --head /tmp/head.json

Where a free model and a free server belong

This split is optional. The pack runs without it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. Use them as a scratch environment, not as the owner of the catalog. A practical split looks like this:

1. On the free server, run the recorder, the property tests, the fixture lock, and the freeze checker against the agent branch.
2. Ask the free model only to propose candidate negative payloads. Write those into failure_modes/quarantine/, which CI ignores.
3. A human promotes a candidate into fixtures/ and adds a catalog row. That promotion is the oracle update.

The model does not patch catalog.json. The server does not need to be your protected CI. It is a place to replay known failures before you spend a gated runner. If you do not want a third-party workspace, run the same commands locally.

Decision table

Observation	Merge?	Next action
All modes failed_as_specified, lock hash match, freeze intact	Yes	Review the implementation diff as usual
wrong_exception on any mode	No	Restore the type, or update the catalog in a human commit
unexpected_success	No	The negative path disappeared; treat as a behavior change
Fixture digest changed	No	Revert fixtures, or accept a human lock update
Frozen test deleted	No	Restore the test; file a flake ticket
Freeze expired	No	Re-triage; do not auto-extend
New quarantine files only	Yes, if catalog unchanged	Review proposals after merge

Limitations

The catalog is only as good as the failures you already named. It will not catch a new class of bug the fixtures never encoded. Property checks over a stale taxonomy create a false ceiling.

Do not use this approach when:

• Failure is intentionally non-deterministic (chaos tests, best-effort telemetry).
• Error payloads cannot leave the trust boundary, and you were about to replay them on a shared free server.
• The codebase has no stable exception types, only stringly {"error": "..."} maps that product wants to reword weekly.
• You need a security audit of the patch. A pinned 403 is not proof that authorization still holds for every object.

YAML freeze files also rot. If nobody triages expiry, the gate turns into a skip list. Cap the freeze window in calendar days and fail closed.

What this does not replace

Hidden tests, mutant scoring, and suite-delta budgets answer other questions. This pack answers one: did the patch move the failures you already named?

Keep the agent's hands off that answer. Then the happy-path suite can go back to being a speed check, not a proof. If you already use a free-model workspace, put failure_modes/ on a path that session cannot write.