Do Not Merge Agent Code Until Its New Tests Kill a Hunk Mutant

A green CI job on an agent patch is not a quality signal. It is a coverage signal for whatever already existed, plus whatever the agent just added. If the new tests cannot kill even one mutant inside the production hunk, the patch has not been checked. Stop the merge.

This is a measurement problem, not a model-quality debate. Agent diffs fail in a specific way: they often extend the suite with assertions that rest on the same control flow the patch just introduced. The suite goes green. The hunk is still untested in any adversarial sense.

What to measure instead

Measure a hunk-local kill ratio. Restrict mutants to the production lines the agent changed. Run only the tests the agent added or modified. Count kills, survivors, and invalid mutants. Ignore the rest of the repository's historical suite for this gate. That older suite can stay as a regression net. It should not be allowed to launder a weak new test file.

A kill, in this gate, means a mutant of the patched hunk made an agent-authored test fail. A survivor means every new test still passed. An invalid mutant means the tree no longer imported or the test harness crashed before assertions. Invalids are discarded, not counted as kills.

Label the numbers below as a worked example, not a field study. Suppose the agent touched 40 production lines and added 3 tests. You apply 12 operators to that hunk. 2 mutants are invalid, 9 survive, 1 is killed. Kill ratio = 1 / 10 = 0.10. That is a merge block under the table in the next section. The suite was green the entire time.

Decision table

Use integer thresholds. Do not negotiate them per model vendor.

Agent test delta	Valid mutants in hunk	Kills	Action
0 new/changed tests	n/a	n/a	Block. No new oracle.
> 0	0 valid	0	Block. Mutator produced nothing usable; widen operators or shrink the hunk.
> 0	1–4	0	Block. Too little pressure.
> 0	≥ 5	0	Block. Tests did not notice the hunk.
> 0	≥ 5	≥ 1 and kill ratio < 0.2	Hold for review. Print survivors next to the diff.
> 0	≥ 5	kill ratio ≥ 0.2	Pass this gate only. Other gates still apply.

The 0.2 floor is a policy choice for a teaching workflow. Raise it for parsers, codecs, and money paths. Lower it only when the hunk is a comment-only or import-only change, which should not have reached this gate at all.

Split the diff before you mutate

Do not mutate the whole tree. Do not run the whole suite. Both hide the thing you need to know.

1. Capture the merge-base and the agent commit.
2. Split production hunks from test hunks.
3. Refuse the patch if the test hunk is empty.
4. Build a file list for mutants from production paths only.
5. Run pytest with an explicit node list taken from the test hunk.

#!/usr/bin/env bash
set -euo pipefail
BASE="${1:-origin/main}"
HEAD="${2:-HEAD}"

git diff --name-only "$BASE"..."$HEAD" > /tmp/agent_files.txt
awk '/(^|\/)tests\/|(^|\/)test_.*\.py$/ {print}' /tmp/agent_files.txt > /tmp/agent_tests.txt
awk '!/(^|\/)tests\/|(^|\/)test_.*\.py$/ {print}' /tmp/agent_files.txt > /tmp/agent_prod.txt

if [[ ! -s /tmp/agent_tests.txt ]]; then
  echo "merge-block: agent produced no test delta" >&2
  exit 2
fi
if [[ ! -s /tmp/agent_prod.txt ]]; then
  echo "merge-block: no production hunk to mutate" >&2
  exit 2
fi

git diff -U0 "$BASE"..."$HEAD" -- $(cat /tmp/agent_prod.txt) > /tmp/agent_prod.hunk

If your layout does not use a tests/ directory, replace the awk filters with your own path policy. The policy has to be written down. An agent that drops tests next to production files will otherwise classify those files as production and mutate the oracle.

A bounded mutator, not a research fuzzer

Full mutation testing tools exist. This artifact is smaller on purpose. It only flips a few operators inside the production hunk, then restores the file. It is a gate sketch. It is not a replacement for mutmut, cosmic-ray, or a typed IR mutator.

# proposal: hunk_mutator.py — teaching artifact, not a production fuzzer
from __future__ import annotations

import pathlib
import re
import subprocess
import sys

OPS = [
    (re.compile(r"(?<![<>=!])==(?!=)"), "!="),
    (re.compile(r"!="), "=="),
    (re.compile(r"\s<\s"), " > "),
    (re.compile(r"\s>\s"), " < "),
    (re.compile(r"\s\+\s"), " - "),
    (re.compile(r"\s-\s"), " + "),
    (re.compile(r"\band\b"), "or"),
    (re.compile(r"\bor\b"), "and"),
    (re.compile(r"True"), "False"),
    (re.compile(r"False"), "True"),
]


def load_hunk_lines(hunk_path: pathlib.Path) -> dict[pathlib.Path, set[int]]:
    current = None
    mapping: dict[pathlib.Path, set[int]] = {}
    for raw in hunk_path.read_text().splitlines():
        if raw.startswith("+++ b/"):
            current = pathlib.Path(raw[6:])
            mapping.setdefault(current, set())
        elif raw.startswith("@@") and current is not None:
            plus = raw.split("+")[1].split(" ")[0]
            start = int(plus.split(",")[0])
            mapping[current].add(start)
    return mapping


def mutate_once(src: str, line_no: int, op_index: int) -> str | None:
    lines = src.splitlines(keepends=True)
    if line_no < 1 or line_no > len(lines):
        return None
    pattern, repl = OPS[op_index % len(OPS)]
    candidate = pattern.sub(repl, lines[line_no - 1], count=1)
    if candidate == lines[line_no - 1]:
        return None
    lines[line_no - 1] = candidate
    return "".join(lines)


def run_agent_tests(nodeids: list[str]) -> str:
    proc = subprocess.run(
        [sys.executable, "-m", "pytest", "-q", *nodeids],
        capture_output=True,
        text=True,
    )
    if proc.returncode == 0:
        return "survive"
    if proc.returncode == 1:
        return "kill"
    return "invalid"

Drive it from a loop that never leaves a dirty tree:

python hunk_mutator.py \
  --hunk /tmp/agent_prod.hunk \
  --tests $(cat /tmp/agent_tests.txt) \
  --report /tmp/kill_ratio.json

Record kills, survivors, invalids, operator name, file, and line. Print survivors in the PR body. A surviving == to != flip on a line the agent just wrote is the whole point of the gate. If reviewers cannot see that line, the number is theater.

Run the campaign where skipping it is expensive

Mutation loops are dull and easy to skip on a laptop. That is the failure mode. If the gate is optional in local pre-push, it will be skipped the moment the agent produces a large hunk.

A remote job that fail-closes on a missing report is the minimum. The job must upload /tmp/kill_ratio.json as an artifact and refuse merge when the file is absent, not only when the ratio is low. Missing evidence is a block, same as a zero kill ratio.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you need a spare runner so the mutation loop is not competing with the editor, MonkeyCode's free model access and free server option can host that job. The merge rule does not change. The model does not get a vote. It may draft extra assertions; those assertions still have to kill a hunk mutant before they count.

Keep the prompt to any model narrow. Ask for additional assertions against a frozen fixture, not for a rewrite of the production hunk. Then feed those assertions through the same kill-ratio gate. Generated tests that cannot kill a mutant are discarded. Do not merge them as documentation of intent.

What the ratio does not mean

A kill ratio of 0.2 is not a proof of functional correctness. It is evidence that at least some agent-authored tests are coupled to the patched lines. Coupling can still be wrong. A test can kill a mutant by asserting an accidental internal, then miss the user-visible contract.

The operator set above does not touch string prefixes, off-by-one on slices, mutated timeouts, or swapped dictionary keys. Survivors against this set can still be real bugs. Invalid mutants can also hide a brittle import graph. If more than about a third of mutants are invalid in a short run, stop and fix collection instead of lowering the floor.

Do not average this ratio with the historical suite. An old test that kills a mutant in the new hunk is useful regression data. It is not evidence that the agent added an oracle. Credit that kill to the old suite, in a separate column, if you want it at all.

Numbered merge workflow

1. Reject any agent PR whose test delta is empty.
2. Compute production hunks at U0 so line numbers stay tight.
3. Collect pytest nodeids from the test delta only.
4. Run the unmutated tree first. If the new tests fail, stop. You do not yet have a baseline.
5. Apply one operator per run. Restore the file after every run.
6. Drop invalids. Compute kill ratio on valid mutants only.
7. Apply the decision table. Paste survivor lines into the review.
8. If the ratio passes, run the full historical suite as a second, separate job.

Step 4 matters. Teams sometimes mutate first and treat a crash as a kill. That inflates the ratio and rewards patches that make the module unimportable under trivial edits.

Limitations

This workflow assumes deterministic unit tests and a language you can parse with cheap text operators. Python comparison flips are a start. They are a bad fit for SQL strings, protobuf wire changes, CSS, and anything whose correctness lives in another process.

It also assumes you can name the test delta. Monorepos that generate tests at build time will need a manifest. Without a manifest, agents will hide tests in generated folders and this gate will either mutate them or miss them.

Flaky tests break the accounting. A mutant that fails 1 run in 5 is not a kill. Quarantine flakes before this gate, or the ratio oscillates and reviewers start ignoring it. Time-dependent tests, network tests, and unordered-set assertions belong outside this loop.

The free runner does not remove those limits. It only removes the excuse that the loop was too slow to run. It does not add model names, quotas, or a claimed accuracy number to the gate. If the remote job cannot restore the tree after a mutant, you will ship dirty artifacts. Add a git checkout -- and a working-tree dirty check at the end of every iteration.

Who should not use this

Do not use this gate as a substitute for a human review on authentication, payments, or privacy-sensitive diffs. A mutant kill can be produced by an assertion on a log line. That is the wrong oracle for those paths.

Do not use it on patches whose production hunk is generated code you do not own. Mutating a vendor file mostly measures whether the generator is brittle.

Do not use it if you cannot freeze fixtures. Property checks with fresh random seeds across mutant runs will mix seed noise into kill counts. Lock the seed, or leave properties out of this particular loop.

Skip the gate for pure deletions of dead code when the test delta only removes tests. That case needs a coverage or reachability check, not a mutator.

The usable output is small: a JSON report, a block/hold/pass bit, and a list of surviving lines. If a patch cannot produce that report, it is not ready. Green was never the question.