A billing pull request lands with thirty-eight generated tests and lifts line coverage from 61% to 88% inside one afternoon. Two weeks later an invoice with a null currency code reaches production, and the first question in the incident thread is how such a dense suite missed it. The answer is usually visible in the diff: the test that covered the currency branch asserts assert invoice.currency is not None and stops there. AI-assisted testing is moving from conference demo to team policy right now, which makes this a good week to be precise about what that policy actually buys.
The three myths below show up repeatedly in review comments on generated tests. None of them is stupid, and each contains a piece of truth that makes the wrong conclusion feel safe. After each correction there is a number you can track per pull request instead of a vibe.
Myth 1: Coverage Is Risk
A coverage report records which lines executed, not which lines were interrogated. Take a function that applies a tiered discount and returns a total; a test calls it once with a large order and asserts assert total > 0. That test executes every line of the function and still passes when the tier comparison is inverted, because a wrong discount produces a positive number too. The coverage tool credits it with full support for the branch, which is why the jump from 61% to 88% feels like progress while detecting almost nothing new.
The corrected mental model is detection rate: of the small, plausible edits a person could make to this code, what fraction does the suite reject? That question is answerable with a script, a timeout, and a diff, and it stays honest even when a model writes confident but shallow assertions. Coverage remains useful as a map of untouched code; it is simply not a proxy for risk.
Myth 2: One Manual Mutation Proves the Test
Breaking a line by hand and watching the suite go red is a real signal, and it is also a sample of one from a very large space. People who do this tend to choose the mutation they already know the answer to, which is precisely the mutation the test was written against. A generated test and a hand-picked manual check can therefore agree with each other completely while both miss the boundary sitting one line below.
The correction is mechanical rather than clever: enumerate the mutation classes that matter for your language, apply them only to lines the pull request touched, and read the survivors. Comparison swaps, boundary shifts, arithmetic swaps, and boolean flips catch most shallow assertions in practice. You are not trying to prove the suite is complete; you are looking for lines where nobody would notice a change.
Myth 3: Mutation Runs Are Too Slow for CI
Full-repository mutation runs are genuinely slow, and quoting a full-run number in order to skip the technique is a category error. The bounded version costs at most max_mutants × test_timeout, and you set both numbers yourself. If your whole suite takes ninety seconds but the affected module runs in eight, twenty mutants against a scoped command cost about three minutes of wall clock in the worst case, and most mutants die in the first second.
A disposable environment helps here for a boring reason: a mutation screen copies the repository, edits one file, runs the suite, and throws the copy away, twenty times in a row. That is workload you want on a box you can delete rather than on the laptop you are also typing on. It is also the case where an ephemeral server stops being a limitation and starts being the point.
A Runnable Screen
The script below uses only the standard library and has no opinion about your framework. It reads a git diff, collects added lines in Python files, applies one operator-class mutation per line, and runs your test command against a temporary copy of the repository. Save it as mutate.py at the repository root.
#!/usr/bin/env python3
'''mutate.py: a bounded, diff-scoped mutation screen. Standard library only.'''
import argparse, dataclasses, pathlib, re, shutil, subprocess, sys, tempfile
OPERATORS = [('==', '!='), ('!=', '=='), ('<=', '>='), ('>=', '<='),
(' + ', ' - '), (' - ', ' + '), (' and ', ' or '), (' or ', ' and '),
('True', 'False'), ('False', 'True')]
def added_lines(diff_text):
targets, path, lineno = {}, None, 0
for line in diff_text.splitlines():
if line.startswith('+++ b/'):
path = line[6:]
targets.setdefault(path, set())
elif line.startswith('@@'):
match = re.search(r'\+(\d+)', line)
lineno = int(match.group(1)) if match else 0
elif path and line.startswith('+') and not line.startswith('+++'):
targets[path].add(lineno)
lineno += 1
elif path and not line.startswith('-'):
lineno += 1
return {p: lines for p, lines in targets.items() if p.endswith('.py') and lines}
@dataclasses.dataclass
class Mutant:
path: str
lineno: int
before: str
after: str
def pick(repo, targets, limit):
found = []
for rel, linenos in targets.items():
source = (repo / rel).read_text(encoding='utf-8').splitlines()
for n in sorted(linenos):
if not 1 <= n <= len(source):
continue
original = source[n - 1]
for old, new in OPERATORS:
if old in original:
found.append(Mutant(rel, n, original, original.replace(old, new, 1)))
break
return found[:limit]
def verdict(mutant, repo, test_cmd, timeout):
with tempfile.TemporaryDirectory() as tmp:
work = pathlib.Path(tmp) / 'repo'
skip = shutil.ignore_patterns('.git', '__pycache__', '.venv', 'node_modules')
shutil.copytree(repo, work, ignore=skip)
target = work / mutant.path
lines = target.read_text(encoding='utf-8').splitlines(keepends=True)
lines[mutant.lineno - 1] = mutant.after + '\n'
target.write_text(''.join(lines), encoding='utf-8')
try:
proc = subprocess.run(test_cmd, shell=True, cwd=work, timeout=timeout,
capture_output=True, text=True)
except subprocess.TimeoutExpired:
return 'timeout'
return 'killed' if proc.returncode != 0 else 'survived'
def main():
parser = argparse.ArgumentParser()
parser.add_argument('--repo', default='.')
parser.add_argument('--base', default='origin/main')
parser.add_argument('--test-cmd', default='python -m pytest -q')
parser.add_argument('--max-mutants', type=int, default=20)
parser.add_argument('--timeout', type=int, default=60)
args = parser.parse_args()
repo = pathlib.Path(args.repo).resolve()
diff = subprocess.run(['git', '-C', str(repo), 'diff', '-U0', args.base],
capture_output=True, text=True, check=True).stdout
tally = {'killed': 0, 'survived': 0, 'timeout': 0}
for mutant in pick(repo, added_lines(diff), args.max_mutants):
result = verdict(mutant, repo, args.test_cmd, args.timeout)
tally[result] += 1
print(f'{result:9} {mutant.path}:{mutant.lineno} {mutant.before.strip()}')
killed, survived = tally['killed'], tally['survived']
scored = killed + survived
if scored:
print(f'\ndetection rate: {killed}/{scored} killed ({100 * killed / scored:.0f}%)')
sys.exit(1 if survived else 0)
if __name__ == '__main__':
main()
Two honest caveats about the implementation. The operator table is a screen rather than a mutation framework, so a survivor may be an equivalent mutant that no test could ever kill; and the copy excludes .git, which breaks tests that shell out to git. Both limitations are cheap to live with once you route survivors through triage instead of treating the exit code as a verdict.
Running It on One Pull Request
Scope the test command to the module that covers the diff, because the timeout budget is per mutant and the wall clock is the product of the two.
git switch -c try/mutation-screen
git diff -U0 origin/main | head -40 # confirm the diff is what you expect
python mutate.py --base origin/main \
--test-cmd 'python -m pytest -q tests/test_billing.py' \
--max-mutants 20 --timeout 60
The output is a list of killed, survived, and timeout lines plus a detection rate. Send the survivor list back to the model that wrote the tests, with the instruction to add the smallest test that kills each one and not to touch the source:
These mutants survived the suite. For each line below, write the minimal
pytest test that fails when the mutation is applied. Assert on returned
values and boundary behaviour, not on the existence of an object.
Generating tests first, screening them second, and re-prompting with survivors turns a one-shot generation into a closed loop with a measurable exit condition. That loop is many short model calls and many short suite runs, which is exactly where free model access and a free server option change the economics of trying it at all.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is described by its operator as an open-source project, and the operator currently advertises free model access, a free ten-million-token allowance, and a free server option. Treat those as time-sensitive commercial terms rather than engineering guarantees, and check the current pricing page before you plan a workflow around them.
If the loop above matches how your team reviews generated tests, it is a reasonable place to run the experiment once and compare the survivor list against your review comments. The comparison, not the marketing page, is the evaluation that matters.
Who Should Not Use This
Teams in regulated domains still need a real mutation framework with full-repository coverage, published configuration, and an auditable report, and a forty-line screen will not satisfy an assessor. Repositories where the affected test module already takes minutes per run will burn the timeout budget on a handful of mutants, so fix the suite first. Projects whose tests depend on docker compose, network fixtures, or git history need the copy step reworked before the numbers mean anything.
None of that weakens the main claim. Coverage tells you where execution went; mutation tells you whether anyone would notice if the behaviour changed. The second question is the one your incident review asks, so it is worth a script.
Top comments (0)