Mutations Catch What Your Agent Patch Misses

Mutations Catch What Your Agent Patch Misses

A date parser broke in production. The agent patch looked correct. All unit tests passed. Two weeks later, a daylight-saving edge case reappeared. The tests never checked the patch's blind spots.

This is the quiet failure mode of AI-generated code. The patch changes behavior silently. The suite stays green. You merge. Then drift shows up months later. The standard remedy is more tests. But tests only cover what the developer knows to check. Agents produce code with unknown unknowns.

Mutation testing attacks that gap. It injects small defects into the source. Then it runs the suite against each mutant. If a test fails, the mutant is killed. If a mutant survives, the suite has a blind spot. Every survivor marks an untested assumption.

I combined mutation testing with property-based checks and a flaky-test freeze. I ran the whole loop on a free server. The setup is reproducible. You can copy it.

Start with a function that parses ISO dates.

from datetime import datetime

def parse_iso(value: str) -> datetime:
    return datetime.fromisoformat(value.replace('Z', '+00:00'))

An agent patch adds support for ISO week dates. It passes the existing tests. But does the new branch have real protection? A plain unit test checks one example.

def test_week_date():
    assert parse_iso('2026-W01-1').year == 2026

That is a single point. Property-based testing explores the space. Hypothesis generates many strings matching the pattern.

from hypothesis import given, strategies as st

@given(st.from_regex(r'[0-9]{4}-W(?:[0-4][0-9]|5[0-3])-[0-9]'))
def test_week_dates_parse_without_error(value):
    parsed = parse_iso(value)
    assert parsed.year >= 2000

This catches format errors. It does not validate calendar correctness. Week 53 of some years does not exist. The property test may miss it.

Now run mutation testing. I used mutmut against the patched file.

$ mutmut run --paths-to-mutate parse.py

The report shows surviving mutants. Each survivor is a place where tests do not care. One survivor changed the week boundary from 53 to 52. No test noticed. The property test generated a range, but the valid week count depends on the year.

I added explicit fixtures for the known boundaries.

FIXED_BOUNDARIES = [
    ('2020-W53-7', True),
    ('2021-W53-7', True),
    ('2026-W53-1', False),
]

The principle is simple. Property tests generate broad input. Fixtures pin rare known values. Mutation testing tells you where the two still miss together.

Flaky tests create another layer of risk. A test that passes sometimes and fails sometimes gives false confidence. I froze flaky tests with a rerun policy.

@pytest.mark.flaky(reruns=3, only_rerun=['AssertionError'])
def test_week_boundary():
    ...

The freeze is explicit. A flaky test does not block the merge. It also does not lie silently. The rerun policy records the evidence.

The whole loop runs every night on a free server. MonkeyCode provides free model access and a free server option. The agent uses the model access to generate candidate patches. The server runs the test loop. I do not provision my own CI hardware for this verification step.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Here is the full loop. First the agent generates a patch. Then the existing suite runs. Property tests follow. Mutation testing comes next. Inspect any surviving mutants. Add fixtures for the missing boundaries. Mark flaky tests with a rerun policy. Repeat until no high-value mutant survives.

The order matters. Property tests find blasts. Mutation tests find blind spots. Fixtures turn blind spots into fixed landmarks. The flaky freeze prevents noise from hiding real failures.

The result is a measurable gate for agent patches. You still need a human to review intent. But mutation testing converts vague trust into a repeatable number.

Who should not do this? Throwaway scripts and short-lived demos do not need the overhead. Mutation testing consumes CPU time. Free servers have resource limits. Use it for code that will be maintained for quarters, not days.

Try this workflow on your next agent patch. Your future self will thank you when the mutation that would have slipped through fails loudly.