Mutate the Fixture, Not the Code: Catching Agent Tests That Only Mirror the Implementation

Most agent-written tests pass for the wrong reason: the fixture already contains the answer. The test asserts that the code reproduces a value that was copied out of the code in the first place. Code mutation testing rarely catches this, because the mutant dies on some other assertion in the same file. Fixture mutation does catch it, and it costs one suite run per mutant.

So the conclusion up front: before you spend effort on property checks or a flake freeze, spend one cheap pass mutating the fixtures and see which tests stop failing. A test that survives a semantic change to its own input is not testing behavior — it is comparing two copies of the same string.

One caveat about everything below: the harness is scaffolding I have not executed in this exact form. Treat the code as a starting point, run it against your own repository, and record your own numbers. Do not inherit mine, because I am not publishing any.

What fixture mutation measures, and what it does not

Fixture mutation is a differential test on the inputs. You perturb a field the code is supposed to consume, rerun the suite, and observe whether anything notices. If nothing notices, one of three things is true: the field is unused, the assertion is too loose, or the test hardcodes the expected output instead of deriving it.

It does not measure coverage, and it does not prove correctness. A mutant can be unkillable for a legitimate reason — a field consumed only on a branch your suite never reaches on purpose. That is exactly why the classification step below has three buckets instead of two.

Step 1: Build a mutant set that is semantically plausible

The cheapest mutants are worthless. Reversing a string or nulling every field produces instant kills and tells you nothing. You want mutations a real upstream system could plausibly send.

# fixture_mutants.py — scaffolding, not executed in this article
import copy, json
from dataclasses import dataclass

@dataclass(frozen=True)
class Mutant:
    id: str
    fixture: str
    pointer: tuple
    op: str
    before: object
    after: object

def walk(node, path=()):
    yield path, node
    if isinstance(node, dict):
        for k, v in node.items():
            yield from walk(v, path + (k,))
    elif isinstance(node, list):
        for i, v in enumerate(node):
            yield from walk(v, path + (i,))

def numeric_drift(doc):
    """Plausible drift: rounding boundaries and quantity edits."""
    for path, node in walk(doc):
        if isinstance(node, int) and not isinstance(node, bool):
            yield path, node, node + 1
        elif isinstance(node, float):
            yield path, node, round(node + 0.01, 2)

def drop_optional(doc):
    """Remove one key at a time; the schema may allow it."""
    for path, node in walk(doc):
        if isinstance(node, dict):
            for k in list(node):
                yield path + (k,), node[k], "__ABSENT__"

def reorder_list(doc):
    """Order-dependence is a real bug class, not a contrived one."""
    for path, node in walk(doc):
        if isinstance(node, list) and len(node) > 1:
            yield path, node, list(reversed(node))

Aim for 15–40 mutants over the fixtures your agent patch actually touches. Bound the set deliberately — the runtime below multiplies by it.

Step 2: Run every mutant N times, including a no-op control

This is the part that makes the result trustworthy when your suite is flaky. Each mutant runs N times with different ordering and hash seeds. The no-op control — a mutant with an identical payload — establishes the baseline instability of the suite itself.

#!/usr/bin/env bash
# run_mutants.sh — scaffolding
set -uo pipefail
OUT=mutants/report.jsonl
: > "$OUT"

for m in mutants/cases/*.json; do
  id=$(basename "$m" .json)
  for seed in 1 2 3; do
    cp "$m" "$FIXTURE_DIR/orders.json"
    PYTHONHASHSEED="$seed" FIXTURE_DIR="$FIXTURE_DIR" \
      pytest -q -p no:cacheprovider tests/ >/dev/null 2>&1
    code=$?
    printf '{"mutant":"%s","seed":%d,"exit":%d}\n' "$id" "$seed" "$code" >> "$OUT"
  done
done

Keep set -e off. You are collecting exit codes, not enforcing success.

Step 3: Classify into three buckets, and exclude harness errors

pytest exit codes carry meaning you must not flatten. 0 is a pass. 1 is a genuine test failure. 2, 3, 4, and 5 are interruption, internal error, usage error, and no-tests-collected — those are harness bugs, not kills. Folding a 5 into the killed bucket is how a mutation report silently becomes fiction.

import collections, json

HARNESS_ERRORS = {2, 3, 4, 5}
rows = [json.loads(l) for l in open("mutants/report.jsonl")]
by = collections.defaultdict(list)
for r in rows:
    by[r["mutant"]].append(r["exit"])

control = by.pop("m000-noop", [])
buckets = {"killed": [], "survivor": [], "unstable": [], "error": []}
for mid, exits in by.items():
    if any(e in HARNESS_ERRORS for e in exits):
        buckets["error"].append(mid); continue
    fails = sum(e == 1 for e in exits)
    passes = sum(e == 0 for e in exits)
    key = "killed" if passes == 0 else "survivor" if fails == 0 else "unstable"
    buckets[key].append(mid)

scored = len(buckets["killed"]) + len(buckets["survivor"])
fms = len(buckets["killed"]) / scored if scored else float("nan")

Report fms next to control and the unstable count, never alone. A score of 0.62 on a suite where the no-op control failed 2 of 3 times means nothing.

Step 4: Read the table, not the number

Head outcome	Base outcome	Interpretation	Action
killed	killed	Assertion already sensitive	Leave it
survivor	killed	Patch removed sensitivity — the strongest signal here	Block the patch
survivor	survivor	Pre-existing gap	File, do not block
unstable	any	Suite instability, not a test-strength result	Freeze ledger
error	any	Harness bug	Fix the runner first

The row that matters for agent patches is survivor / killed. Agents are very good at deleting the one assertion that made a fixture edit fatal, because the deletion makes the new test green. That delta is a regression in verification strength, and it is invisible to a pass/fail gate.

Running the base tree doubles your runtime. If that is unaffordable, run base once per module rather than once per patch, and accept the coarser resolution.

Step 5: Freeze unstable mutants without laundering them into kills

An unstable mutant is a flaky-test finding wearing a mutation costume. It must never be counted as killed, and it must never be counted as a survivor either — it sits outside the denominator with an owner and an expiry.

# freeze-ledger, one entry per unstable mutant
- test: tests/test_orders.py::test_total_rounding
  detected_by: fixture-mutant m012 (numeric_drift)
  observed: 1/3 pass on unmutated fixture
  owner: payments
  opened: 2026-11-06
  expires: 2026-12-06
  exit_criteria: 20 consecutive clean runs across 3 seeds, unmutated fixture

Two rules keep the ledger honest. First, an expired entry fails the build; a freeze that cannot rot is a permanent deletion. Second, a frozen test cannot be cited as evidence for the patch that touched it — otherwise the freeze quietly becomes an approval.

Where a free-tier model genuinely helps

Two parts of this workflow are tedious rather than hard: generating semantically plausible mutants, and triaging survivor clusters. That is where I would use MonkeyCode's free model access — feeding it a fixture plus the consuming function's signature and asking for candidate mutations with a one-line rationale each, then grouping the survivors by which assertion they should have tripped.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free server option matters for a different reason: the mutant matrix is embarrassingly parallel and boring, exactly the workload you do not want consuming your interactive machine. Check the current terms yourself before depending on either capability — I am not asserting quota, hardware, or permanence, and availability at that tier is not something this article can promise. Treat the free tier as a burst resource: run the matrix, export the JSONL, go home. And do not ship private fixtures or production-derived data to any hosted model without your own review.

Keep the model on the generator side, not the judge side. It proposes mutants; the test suite votes. A model that both proposes and evaluates its own mutants reproduces exactly the tautology this workflow exists to catch.

Cost, limits, and who should skip this

Runtime is mutants × seeds suite runs. With 25 mutants and 3 seeds that is 75 runs, which is why you scope the fixture set to files the patch touched rather than the whole repository.

Skip this if any of the following is true:

• Your suite already takes more than a few minutes; you need test-level selection before mutation.
• Your fixtures are generated at runtime from a schema — mutate the generator's inputs instead, and note that the mapping back to a failing test gets fuzzier.
• You have no stable no-op control. Without it, every number below is noise, and the right next task is the flake freeze, not this one.
• Your tests assert on opaque outputs you cannot perturb meaningfully; property checks will serve you better here.

Fixture mutation is not a replacement for property-based testing. It is the cheap prior step that tells you whether property checks would be answering a question anyone is asking. If you run the no-op control against a fixture-heavy suite, the unstable count is the number I would want to compare — it caps how much trust anything else in the report deserves.