Mutate the Agent's Patch Before You Trust Its Tests

An agent's green test run is a statement about the inputs the agent imagined. It says nothing about the inputs it forgot. Mutation testing measures that gap directly: you break the patch on purpose, run the tests, and count how many breaks go unnoticed.

I keep seeing the same pattern in agent-generated C++: the patch is plausible, the tests pass, and the suite still cannot fail on a wrong implementation. The fix is not more tests. The fix is a gate that tests the tests.

Why passing tests are not evidence

A test suite has value only if it fails when the code is wrong. That property is not implied by "all tests pass." It has to be demonstrated.

For human-written code, we usually trust the author's intent. For agent patches, intent is the least reliable input. The model writes the implementation and the tests from the same context, so the tests tend to encode the same assumptions as the code. When the implementation is wrong in a way that matches its own assumptions, the suite is blind by construction.

Mutation testing is the cheapest way to expose that blindness.

The gate: five steps

Run this after the agent's tests pass, before you merge.

Step 1 — Baseline. Run the agent's tests against the unpatched code. If the patch fixes a bug, some tests should fail here. If it is a refactor, they should pass. Record the result.

Step 2 — Apply the patch. The tests must pass. If they do not, stop. Send the failing suite back to the agent instead of debugging its code yourself.

Step 3 — Generate mutants. Apply one small semantic change to the patch at a time. Flip a comparison. Drop a guard. Change find to find_last_of. Remove an atomic store. Change a memory order. One change per mutant, never two.

Step 4 — Run the tests against each mutant. If at least one test fails, the mutant is killed. If the whole suite stays green, the mutant survived. A survivor is a hole in the test suite.

Step 5 — Score and decide. Mutation score = killed mutants / total mutants. Set the merge threshold before you start, not after. A reasonable starting point is 80% on the patch's own logic; below that, send the tests back to the agent with the survivor list attached.

A minimal reproduction

Here is the smallest case that shows the pattern. The agent's patch parses key:value lines:

// kv.hpp — the agent's patch
#pragma once
#include <string>

bool parse_kv(const std::string& line, std::string& key, std::string& value) {
    auto colon = line.find(':');
    if (colon == std::string::npos) return false;
    key = line.substr(0, colon);
    value = line.substr(colon + 1);
    return true;
}

The agent also wrote a test:

// kv_test.cpp — the agent's test
#include <gtest/gtest.h>
#include "kv.hpp"

TEST(ParseKV, SplitsOnFirstColon) {
    std::string key, value;
    ASSERT_TRUE(parse_kv("mode:fast", key, value));
    EXPECT_EQ(key, "mode");
    EXPECT_EQ(value, "fast");
}

Green. Now mutate the patch:

Mutant	Change	Result
M1	find(':') → find_last_of(':')	survived
M2	delete the npos guard	survived
M3	substr(colon + 1) → substr(colon)	killed
M4	substr(0, colon) → substr(0, colon + 1)	killed
M5	find(':') → find('=')	killed

Score: 3/5 = 60%. The two survivors are exactly the interesting bugs. M1 changes the split point on multi-colon input. M2 turns malformed input into a silent success. The agent's single happy-path test cannot detect either.

The survivors tell you which tests to add:

TEST(ParseKV, SplitsAtFirstColonNotLast) {
    std::string key, value;
    ASSERT_TRUE(parse_kv("a:b:c", key, value));
    EXPECT_EQ(key, "a");
    EXPECT_EQ(value, "b:c");
}

TEST(ParseKV, RejectsLineWithoutColon) {
    std::string key, value;
    EXPECT_FALSE(parse_kv("no-colon-here", key, value));
}

Now 5/5. The suite can fail on a wrong patch. That is the property you actually wanted.

A minimal harness

The gate is a loop: apply mutation, rebuild, run tests, restore. A small script is enough to start. This one is deliberately naive — regex-based, one file at a time:

#!/usr/bin/env python3
"""mutate_gate.py — run the test suite against each mutant of a patch."""
import re, subprocess, sys
from pathlib import Path

MUTATIONS = [
    ("find->find_last_of", r"find\(':'\)", r"find_last_of(':')"),
    ("drop-npos-guard", r"if \(colon == std::string::npos\) return false;\n", ""),
    ("value-includes-colon", r"substr\(colon \+ 1\)", r"substr(colon)"),
    ("key-includes-colon", r"substr\(0, colon\)", r"substr(0, colon + 1)"),
    ("find->find-eq", r"find\(':'\)", r"find('=')"),
]

def run(cmd, cwd):
    return subprocess.run(cmd, shell=True, cwd=cwd,
                          capture_output=True, text=True).returncode == 0

def main():
    src = Path(sys.argv[1])
    build_cmd, test_cmd = sys.argv[2], sys.argv[3]
    original = src.read_text()
    print(f"pre-mutation tests pass: {run(test_cmd, src.parent)}")

    survived = []
    for name, pattern, replacement in MUTATIONS:
        src.write_text(re.sub(pattern, replacement, original))
        built = run(build_cmd, src.parent)
        passed = run(test_cmd, src.parent)
        status = "SURVIVED" if built and passed else "killed"
        if status == "SURVIVED":
            survived.append(name)
        print(f"{name}: {status}")
        src.write_text(original)

    total = len(MUTATIONS)
    score = 1 - len(survived) / total
    print(f"mutation score: {score:.0%} ({total - len(survived)}/{total})")
    if survived:
        print("survivors to cover:", ", ".join(survived))

if __name__ == "__main__":
    main()

Run it like this:

python3 mutate_gate.py kv.hpp "cmake --build build" "ctest --test-dir build"

This is a starting point, not a tool. For production C++, use a real mutation engine such as Mull. The script exists to make the gate cheap enough to run on every agent patch, including throwaway ones.

Where this pays off for agent patches

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The gate is model-agnostic. In my setup, the candidate patch and its tests came from MonkeyCode's free model access, and the free server option was the environment for the iteration loop. The mutation gate itself is just a compiler and a test runner; it does not care which model wrote the patch.

That is the point. Agents optimize for green tests. The mutation gate changes the success criterion from "the tests pass" to "the tests can fail." The second property is the one that survives contact with the next refactor.

The economics work out because the gate is scoped. You mutate the patch's own logic, not the whole codebase. A five-mutant run is five rebuilds and five test invocations. That is minutes, not hours.

Limitations

Mutation testing finds missing assertions. It does not find races, undefined behavior, or logic that is wrong but internally consistent.

• Equivalent mutants. Some mutants behave identically to the original. Review survivors before writing tests; not every survivor is a real gap.
• Cost scales with mutants. Each mutant is a rebuild plus a test run. Keep the scope to the patch's changed lines. If the test suite cannot be scoped, this gate is too slow to run per patch.
• Regex mutations are fragile. The harness above is for demonstration. On a real codebase, use a tool that operates on the AST, or you will spend your time debugging the mutator instead of the code.
• No concurrency coverage. Mutation testing will not catch a missing atomic or a wrong memory order the way ThreadSanitizer will. Run TSan after the mutation gate, not instead of it.

Who should not use this

Three cases where the gate is the wrong tool.

• Throwaway scripts. If the tests are deleted with the code, the survival rate does not matter.
• Hour-long suites. If you cannot run a scoped subset, the gate becomes a nightly job and loses its feedback loop.
• Teams that will not review survivors. An unscored mutation run is noise. A scored run that nobody reads is worse — it manufactures false confidence.

The takeaway

The next time an agent reports green tests, ask one question: what would make those tests fail? If you cannot answer, mutate the patch and find out. The survival rate is the number that matters, not the pass rate.

I would be curious what your survival rate looks like on agent patches — the comments are open.