Treat a Mutant Kill Map as the Merge Token for Agent Patches

A green test run after an agent patch is a weak merge token. The new tests may never have been able to fail. The stronger artifact is a kill map: which seeded mutants died at HEAD, which died after the patch, and which came back to life.

That delta is reviewable. A coverage percentage is not.

The failure mode

Agent patches often arrive with extra tests. Those tests tend to assert the new code's own return values. They compile. They pass. They do not prove that a regression would turn red.

Three patterns show up in review queues.

1. The agent adds a test that reconstructs the implementation.
2. The agent extends a fixture until the new branch is the only path that runs.
3. The agent parks an unstable check in skip, xfail, or a longer timeout so the suite stays green.

Property checks, fixture hashes, and a freeze on flaky tests are the controls. The kill map is how you measure whether those controls detected anything.

What a kill map records

Store one JSON document per revision. Do not treat it as a badge. Treat it as an inventory of faults the suite can still detect.

{
  "rev": "HEAD",
  "seed": "mutants-v1",
  "killed": ["pred_negated:pricing.py:88", "off_by_one:queue.py:41"],
  "survived": ["arg_swap:cache.py:17"],
  "unexecuted": ["timeout_path:worker.py:204"],
  "fixture_sha256": "9c1e…",
  "flake_freeze_sha256": "b3aa…"
}

Killed means a test failed under that mutant. Survived means every collected test still passed. Unexecuted means the mutant sat outside the collected set. The two hashes freeze the fixture catalog and the flake quarantine file.

If the agent rewrites tests but the killed set shrinks, the patch deleted detection. It did not remove noise.

Proposed workflow

The steps below are a merge policy, not a measured production report. Label every generated mutant as synthetic. Do not publish a mutation score you have not computed on your tree.

1. Freeze fixtures and the flake list at HEAD

Hash the fixture directory and the quarantine file before the agent runs. Record both digests in the HEAD kill map.

find tests/fixtures -type f -print0 | sort -z | xargs -0 sha256sum > /tmp/fixtures.list
sha256sum /tmp/fixtures.list tests/flake_freeze.toml

An agent may add a fixture only with a provenance header. It may not rewrite an existing file. It may not append to flake_freeze.toml.

Proposed fixture header:

# provenance: recorded | synthesized | mutated-from:<path>
# owner: <reviewer>
# oracle: <property or assertion that can fail>

A synthesized fixture cannot be the sole passing input for a new test. Pair it with one recorded or mutated-from fixture, or reject the test file.

2. Seed a small, named mutant catalog

Keep the operator set tiny and deterministic. Three operators are enough to start: negate a boolean predicate, shift an integer literal by one, swap the first two arguments of a call.

# proposed: seed_mutants.py — synthetic AST mutants, not a published score
import ast, pathlib

class MutantLister(ast.NodeVisitor):
    def __init__(self, path):
        self.path = path
        self.found = []

    def visit_Compare(self, node):
        self.found.append(
            {"op": "pred_negated", "file": self.path, "line": node.lineno}
        )
        self.generic_visit(node)

    def visit_Constant(self, node):
        if isinstance(node.value, int):
            self.found.append(
                {"op": "off_by_one", "file": self.path, "line": node.lineno}
            )
        self.generic_visit(node)

    def visit_Call(self, node):
        if len(node.args) >= 2:
            self.found.append(
                {"op": "arg_swap", "file": self.path, "line": node.lineno}
            )
        self.generic_visit(node)

def list_mutants(root="src"):
    out = []
    for p in pathlib.Path(root).rglob("*.py"):
        tree = ast.parse(p.read_text())
        visitor = MutantLister(str(p))
        visitor.visit(tree)
        out.extend(visitor.found)
    return out

Cap the catalog. A useful default is the files in the agent's diff plus direct importers. Whole-monorepo seeding turns the gate into a timeout.

Version the operator list in git. If the agent can edit seed_mutants.py, the ledger is fiction.

3. Write properties against retry, auth, and serialization surfaces

Line coverage on the edited helper is the wrong target. Mutants usually survive there. They die on callers, privilege checks, retry state, and encoded output.

Keep those properties independent of the agent's new names.

# proposed property: balances stay non-negative after any retry sequence
from hypothesis import given, strategies as st

@given(st.lists(st.integers(min_value=-5, max_value=50), min_size=1, max_size=8))
def test_retry_never_goes_negative(amounts):
    ledger = Ledger()
    for n in amounts:
        ledger.record(n)
        ledger.retry_last_if_failed()
    assert ledger.balance() >= 0

A second property should pin authorization, not arithmetic.

def test_foreign_account_write_is_rejected(foreign_id):
    session = Session(user="alice")
    try:
        session.transfer(to=foreign_id, amount=1)
    except PermissionError:
        assert session.audit()[-1].denied
        return
    raise AssertionError("foreign write succeeded")

If a property only imports the function the agent just wrote, it is likely restating the patch. Move the oracle to a surface the agent was not asked to rename.

4. Run the catalog twice: HEAD, then the patch

Execute each mutant in isolation. One process per mutant. Capture fail versus pass versus not-collected.

python seed_mutants.py --rev HEAD --out killmap-head.json
# apply agent patch
python seed_mutants.py --rev PATCH --out killmap-patch.json
python compare_kill_maps.py killmap-head.json killmap-patch.json

A leased runner helps. Mutant jobs pollute local caches and shared CI workspaces if they share site-packages.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option. The model access is useful for drafting extra property sketches that a reviewer still has to accept by hand. The free server is useful as an isolated place to apply mutants and collect the two kill maps without contaminating the main pipeline.

Do not let the model edit flake_freeze.toml or rewrite fixture bytes. Those files are inputs to the gate, not outputs of the agent.

5. Compare maps with explicit fail reasons

# proposed: compare_kill_maps.py
import json, sys

def load(path):
    with open(path) as handle:
        return json.load(handle)

head, patch = load(sys.argv[1]), load(sys.argv[2])
head_k, patch_k = set(head["killed"]), set(patch["killed"])
resurrected = sorted(head_k - patch_k)
new_kills = sorted(patch_k - head_k)

errors = []
if resurrected:
    errors.append(f"resurrected mutants: {resurrected}")
if patch["fixture_sha256"] != head["fixture_sha256"]:
    if not patch.get("declared_new_fixtures"):
        errors.append("fixture catalog changed without provenance")
if patch["flake_freeze_sha256"] != head["flake_freeze_sha256"]:
    errors.append("flake freeze file mutated")
if not new_kills and patch.get("new_tests"):
    errors.append("new tests added but no additional mutant died")

print("\n".join(errors) or "kill map delta acceptable")
sys.exit(1 if errors else 0)

New tests with zero new kills are coverage theater. A resurrected mutant is a detection regression. A changed flake freeze is an unearned green.

Decision table

Observation	Merge
New tests, no new kills	Reject
Killed set shrinks	Reject
Fixture bytes change, no provenance	Reject
flake_freeze.toml digest changes	Reject
New kills, fixtures declared, freeze unchanged	Allow review
Survived mutants only in untouched files	Record, do not block

The last row matters. Surviving mutants outside the edited files are backlog. They are not a veto.

Flaky freeze rules the agent cannot negotiate

Quarantine is a human artifact. Give each frozen test an owner, a reason, and an expiry date.

# tests/flake_freeze.toml — humans only
["tests/test_billing.py::test_webhook_order"]
owner = "billing-oncall"
reason = "duplicate delivery from vendor sandbox"
expires = "2026-09-25"

The gate should refuse any agent diff that:

1. Adds a name to the freeze file.
2. Extends an expiry.
3. Converts a failure into skip, xfail, or a raised timeout.

When a freeze expires, the test returns to the kill-map run. If it is still flaky, a person updates the freeze. The agent does not.

Timeouts need the same treatment as skips. A patch that lifts timeout=2 to timeout=30 on an existing test is a freeze in disguise. Diff the pytest markers and fail the compare step when they widen.

What belongs in the pull request

Paste both kill maps and the compare script output. That is the merge token. The green check is supporting evidence, not the artifact.

Also attach:

1. The fixture provenance headers for any new files.
2. The unchanged flake_freeze.toml digest.
3. The operator seed version (mutants-v1 in the example).

Reviewers can then ask one question: which new mutant died, and which old mutant is still dead?

Limitations

This policy does not compute a scientific mutation score. Equivalent mutants will survive. AST operators miss concurrency bugs, clock skew, and I/O faults. Property checks are only as good as the strategies you write.

Model-drafted sketches are untrusted. They can propose properties that restate the patch. They can also propose mutants that never compile. A human has to accept both lists before the gate is real.

Runtime grows with catalog size. Isolate mutant processes. Do not run this on a machine that also holds production credentials. Do not treat a single leased server as durable infrastructure. The workflow assumes you can rebuild the catalog from git.

Who should not use this

Skip the kill map if your suite is non-deterministic, depends on live network clocks, or cannot run twice on the same commit. Skip it if review already blocks test-file edits and no agent is writing tests. Skip it for generated protobufs and snapshot-only UI packs until you have an oracle other than byte equality.

Teams that only want a badge will game the seed. Keep the operator list short and versioned. If merge pressure is high, shrink the catalog before you weaken the compare rules.

The merge token is the kill-map delta. Keep pytest green as a precondition, not as the proof.