Canary Mutations for Fixtures: A Sensitivity Gate for Agent-Written Patches

A green suite is the weakest evidence an agent-authored patch can hand you. Gate on fixture sensitivity instead: every fixture a patch touches must still be able to fail something. This post gives you three layers — property obligations, canary mutations, and a freeze registry with an expiry — plus the code to wire them into a pre-merge check.

The short version

Tests pass for two reasons: the code is correct, or the test stopped constraining anything. Agents are very good at the second one without meaning to. They add a field to a fixture, relax an assertion, or widen a type until the new code path fits. Nothing in the diff looks wrong. The suite is still green.

So stop asking "did the tests pass" and start asking "how many deliberate breakages does this fixture still catch?" That number is measurable, cheap to compute, and it moves before the suite does.

Why a passing suite is a lagging indicator

The failure mode is fixture drift. A fixture starts as a minimal, opinionated object: three fields, one nested list, one boundary value. A patch adds a fourth field. Then it adds a default for a missing field. Six patches later the fixture is a superset that satisfies every code path and constrains none of them.

Assertions drift the same way. assert result.ok is a weaker contract than assert result == Expected(items=[...], total=Decimal("41.10")). An agent that swaps the second for the first has not lied to you — it has simplified a test that was making its patch hard to land.

The gate below is designed for that specific dishonesty-free failure. It does not try to detect malice. It measures whether the test harness still bites.

Layer 1: write property obligations before you review the diff

A property obligation is a one-line contract plus the breakage it is supposed to catch. Keep them in a file the patch has to touch, not in a reviewer's head.

# tests/obligations.yaml
- id: invoice-total-is-sum-of-lines
  property: "sum(line.amount for line in invoice.lines) == invoice.total"
  canary: "truncate Decimal to int"
  catches: tests/billing/test_invoice.py::test_total_matches_lines

- id: retry-stops-at-three
  property: "client.attempts <= 3 for any permanent 4xx"
  canary: "remove the attempt counter increment"
  catches: tests/net/test_retry.py::test_gives_up_after_three

- id: empty-cart-has-zero-total
  property: "total == 0 and tax == 0 when lines == []"
  canary: "return None for empty input"
  catches: tests/billing/test_cart.py::test_empty_cart

Three rules keep this file honest.

1. Every entry must name a test that exists. A stale catches: path is a broken obligation, not a TODO.
2. Every entry must name a canary — a concrete edit that should turn the test red. If you cannot describe the breakage in one sentence, the property is too vague to enforce.
3. Entries are reviewed with the diff, not after it. If a patch deletes or renames a catches: target, the obligations file must change in the same commit.

This file is also the cheapest place to put an agent to work: ask it to propose a canary for each new property, then verify the canary yourself by applying it.

Layer 2: prove each fixture still catches its canary

This is the measurement. For each fixture file, apply a small set of mechanical mutations in a scratch copy of the repo, run the tests that depend on that fixture, and count how many mutations the suite notices.

# tools/fixture_canary.py  — stdlib + pytest only; point the paths at your repo
from __future__ import annotations
import argparse, json, re, shutil, subprocess, sys, tempfile
from pathlib import Path

MUTATIONS = {
    # text/JSON/YAML fixtures: blank every scalar value
    "blank_values": lambda t: re.sub(r'(?m)(:\s*)(\S.*)$', r'\1""', t),
    # replace every scalar with null
    "null_out":     lambda t: re.sub(r'(?m)(:\s*)(\S.*)$', r'\1null', t),
    # truncate the file to half its lines (drops records)
    "truncate":     lambda t: "\n".join(t.splitlines()[: max(1, len(t.splitlines()) // 2)]),
}

def caught(repo: Path, fixture: Path, tests: list[str], mut: str, timeout: int = 600) -> bool:
    with tempfile.TemporaryDirectory() as tmp:
        work = Path(tmp) / "repo"
        shutil.copytree(repo, work, ignore=shutil.ignore_patterns(".git", ".venv", "node_modules"))
        target = work / fixture.relative_to(repo)
        target.write_text(MUTATIONS[mut](target.read_text()))
        proc = subprocess.run(
            [sys.executable, "-m", "pytest", "-x", "-q", *tests],
            cwd=work, capture_output=True, text=True, timeout=timeout,
        )
        return proc.returncode != 0

def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--map", default="tests/fixture_usage.json")
    ap.add_argument("--baseline", default="tests/fixture_sensitivity.json")
    ap.add_argument("--min", type=float, default=0.34)
    ap.add_argument("--write-baseline", action="store_true")
    args = ap.parse_args()

    repo = Path.cwd()
    usage = json.loads(Path(args.map).read_text())  # {"tests/fixtures/invoice.yaml": ["tests/billing"]}
    report = {}
    for fixture, tests in usage.items():
        results = {m: caught(repo, Path(fixture), tests, m) for m in MUTATIONS}
        report[fixture] = {
            "caught": sum(results.values()),
            "total": len(results),
            "missed": [m for m, ok in results.items() if not ok],
        }

    if args.write_baseline:
        Path(args.baseline).write_text(json.dumps(report, indent=2) + "\n")
        return 0

    base = json.loads(Path(args.baseline).read_text()) if Path(args.baseline).exists() else {}
    failed = False
    for fixture, r in report.items():
        ratio = r["caught"] / r["total"]
        floor = base.get(fixture, {}).get("caught", r["total"])
        if ratio < args.min or r["caught"] < floor:
            failed = True
            print(f"WEAK FIXTURE {fixture}: caught {r['caught']}/{r['total']} "
                  f"(floor {floor}) missed={r['missed']}")
    return 1 if failed else 0

if __name__ == "__main__":
    raise SystemExit(main())

The fixture_usage.json map is the only manual part. Generate it once with a grep for fixture paths in your test tree, then review it by hand — a wrong mapping produces false confidence in both directions.

Run it in two modes:

# first time: record current sensitivity as the floor
python tools/fixture_canary.py --write-baseline

# every PR: fail if any fixture got weaker
python tools/fixture_canary.py --min 0.34

The baseline is the point. Absolute sensitivity is noisy across fixtures; the interesting signal is a drop for a fixture the patch touched. A fixture that goes from 3/3 to 1/3 in a patch that also edits the assertions is a loud signal, and it costs you one line in CI.

Layer 3: freeze flaky tests with an owner and an expiry date

Flaky tests poison this gate. If a test fails intermittently, every canary appears caught, and sensitivity is meaningless. So freeze it — but freeze it as a debt with terms.

# tests/quarantine.yaml
frozen:
  - test: tests/net/test_retry.py::test_backoff_timing
    owner: net-team
    until: 2026-11-30
    reason: "wall-clock assertion, needs a fake clock"
    covers_obligations: [retry-stops-at-three]

# tools/check_freezes.py — run on a daily schedule, not on PRs
import datetime, sys, yaml

today = datetime.date.today()
for e in yaml.safe_load(open("tests/quarantine.yaml"))["frozen"]:
    if datetime.date.fromisoformat(e["until"]) <= today:
        print(f"EXPIRED FREEZE {e['test']} owner={e['owner']}")
        sys.exit(1)

Two constraints make the freeze honest. First, a freeze that lists covers_obligations must be excluded from the sensitivity run for those fixtures, or the canary gate will report a pass on a test nobody is running. Second, expiry is a hard failure on a schedule — a freeze without a working alarm is just a deletion with extra steps.

Wiring it into an agent loop

Numbered workflow I use for patch review:

1. Agent opens a patch. Before reading code, read tests/obligations.yaml diff. No obligation changes plus test changes is a red flag.
2. Run fixture_canary.py against the patch branch and compare to the baseline from main.
3. Run check_freezes.py. If an expired freeze is involved, stop — the patch is being evaluated against a test that may not run.
4. Only then read the code diff.

Step 2 is the expensive one, because it multiplies your test runs by the number of mutations per touched fixture. That is exactly the kind of embarrassingly parallel, CPU-only job that a free server option is useful for — the work is short-lived and needs no GPU. It is also where free model access helps: ask a model to propose fixture mutations that are semantically plausible (swap a currency code, flip a sign, change a status enum) rather than mechanical, then verify each proposal by actually applying it. Generated canaries are candidates until you run them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project offering free model access and a free server option; the operator advertises those two things, and any specific token allowance or capacity is promotional and should be checked against the project's current terms before you build a pipeline on it. Treat the free tier as a way to try the workflow, not as a capacity plan.

Decision table

Fixture sensitivity	Obligation diff	Freeze state	Action
Baseline or better	Present, all targets exist	None touching	Merge after normal review
Any drop, patch untouched fixtures	Present	None	Hold — ask why the patch weakened an unrelated fixture
Any drop on touched fixture	Missing	None	Reject — resubmit with an obligation for the new path
1/3 or below	Present	None	Hold — fixture needs a boundary value before the patch lands
Any	Any	Expired freeze in the test path	Reject until the freeze is resolved or re-justified
Any	catches: target deleted	Any	Reject — an obligation cannot outlive its test

Limitations, and who should not do this

This gate does not check correctness. It checks that your harness still reacts. A suite that catches every canary can still codify the wrong behavior, and a fixture at 3/3 sensitivity can encode a business rule nobody wants.

The mutation set is mechanical and shallow. blank_values will not catch a fixture that is sensitive only to a specific numeric boundary, and text-level regex mutation does poorly on binary fixture formats. If your fixtures are generated from a schema, mutate the schema, not the file.

Cost is real. N fixtures × 3 mutations × your test subset is a lot of pytest invocations for a large repo. Start with the ten fixtures your recent patches touched most, not all of them.

Skip this if you are pre-product with fewer than a dozen tests — you have no fixture drift to detect yet. Skip it if your suite is already slower than your review loop; fix the runtime first. Skip it if a single fixture is shared by your entire test tree: the canary result will be one number for the whole repo and tells you nothing locally.

If you want to try the parallel mutation sweep without provisioning a runner, the free model access and free server option in MonkeyCode are a reasonable place to start — bring the ten-fixture subset and the baseline file, and see whether the gate fires on a patch you already know is weak.