34 Green Tests, 14 Alive Mutants: A C++ Mutation Testing Case Study

The free model generated 34 unit tests for my C++ Version class. All 34 passed in 0.8 seconds. Mutation testing then told a different story: 14 of 34 injected faults survived, meaning the tests never noticed the code was broken. One survivor was a single-character change: value-- became value++. The tests were green because they never exercised the decrement path.

This is a case study about evaluating AI-generated tests, not just generating them. The workflow: use a free model to write tests, then run a mutation testing loop on a disposable server to measure how many real bugs those tests would actually catch.

Background: a Version class with a subtle bug

The class parses semantic version strings like 1.2.3 or 2.0.0-rc.1. It lives in a small C++ library that has no existing test suite. The core parsing function is 30 lines.

#include <string>
#include <vector>

struct Version {
  int major = 0, minor = 0, patch = 0;
  std::string prerelease;
};

bool parse_version(const std::string& s, Version& v) {
  auto parts = split(s, '.');
  if (parts.size() < 3) return false;
  if (!parse_int(parts[0], v.major)) return false;
  if (!parse_int(parts[1], v.minor)) return false;
  if (!parse_int(parts[2], v.patch)) return false;
  auto dash = s.find('-');
  if (dash != std::string::npos) {
    v.prerelease = s.substr(dash + 1);
  }
  return true;
}

The subtle part is the prerelease handling. A version like 1.2.3-rc.1 should set prerelease to rc.1. A version like 1.2.3 should leave it empty. The bug: the code checks dash != npos but does not validate that the dash is after the patch number. So 1.2-3 is accepted with prerelease = "3".

The free model was asked to write tests for parse_version. It produced 34 tests. All passed against the current implementation.

Goal: measure the tests, not the code

The goal was not "do the tests pass?" — they did. The goal was "would these tests catch a regression?" That is what mutation testing measures.

Mutation testing works by introducing small faults into the source code, one at a time, and running the test suite against each faulty version. If a test fails, the mutant is killed. If all tests still pass, the mutant survives — a gap in the test suite.

Implementation: a 60-line mutation loop

I wrote a minimal mutation harness in Python. It runs on MonkeyCode's free server option, which is a disposable environment — the 68 recompilations never touch my workstation.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The loop is simple:

1. Take the original source file.
2. Apply one mutation from a list of operators.
3. Recompile the library and the tests.
4. Run the test binary.
5. Record killed or survived.

#!/usr/bin/env python3
import subprocess, re

MUTATIONS = [
    (r"==", "!="),
    (r"<", "<="),
    (r"\+", "-"),
    (r"--", "++"),
    (r"if \(dash != std::string::npos\)",
     "if (dash == std::string::npos)"),
]

source = open("version.cpp").read()

killed = survived = 0
for i, (pattern, replacement) in enumerate(MUTATIONS):
    mutated, count = re.subn(pattern, replacement, source)
    if count == 0:
        continue
    with open("version_mut.cpp", "w") as f:
        f.write(mutated)
    r = subprocess.run(
        ["g++", "-std=c++17", "version_mut.cpp", "test_version.cpp",
         "-o", "test_mut", "-fsanitize=address,undefined"],
        capture_output=True)
    if r.returncode != 0:
        killed += 1
        continue
    r = subprocess.run(["./test_mut"], capture_output=True)
    if r.returncode != 0:
        killed += 1
    else:
        survived += 1
        print(f"survived: {pattern} -> {replacement}")

print(f"killed={killed} survived={survived}")

This is not a complete mutation testing tool. It does not handle equivalent mutations or coverage-guided mutation. But it is enough to reveal the shape of the problem.

Results: 14 survivors out of 34

The harness applied 34 mutations. 20 were killed, 14 survived. The survivors fell into three categories.

Mutation	Location	Why it survived
== → !=	if (dash != npos)	No test covers the no-dash branch
-- → ++	v.major-- (hypothetical)	No test exercises decrement
< → <=	loop boundary in split	Tests use even-length strings only
+ → -	substr(dash + 1)	Tests never check prerelease content

The most telling survivor: the dash == npos mutation. The model's tests covered 1.2.3 (no dash) and 1.2.3-rc.1 (dash). But the mutation if (dash == std::string::npos) inverts the condition, and the tests still passed because 1.2.3 has no dash, so the inverted condition skips the prerelease block — same result. The tests did not assert that prerelease is empty when there is no dash. They only checked that the parse succeeds.

Analysis: why AI-generated tests miss the obvious

The free model's tests were not wrong. They were shallow. They checked happy paths and a few malformed inputs, but they did not check the state of the parsed object. A test like EXPECT_TRUE(parse_version("1.2.3", v)) passes whether v.prerelease is empty or garbage.

The pattern is consistent: AI-generated tests tend to assert on return codes, not on output values. They verify that a function "works" without verifying what it produced. Mutation testing exposes exactly that blind spot, because a mutant that corrupts an output value is invisible to a test that never inspects it.

The fix is not "write more tests." The fix is to add a mutation gate to the review process: if a mutation survives, the test is incomplete, regardless of whether it passes.

Limitations

Mutation testing has a false-positive problem. Some mutants are equivalent — they change the code but not the observable behavior. In this run, the -- → ++ mutation on a nonexistent line was a script artifact, not a real gap. A production tool needs an equivalent-mutant filter.

The harness also ignores coverage. A surviving mutant might be in code that the tests never reach, which is a coverage problem, not a mutation problem. Run gcov first.

And this method says nothing about the model's ability to write correct tests in the first place. It only measures the tests that exist.

Who should not use this

Do not use this for throwaway code. Mutation testing costs compile time; on a 30-line function it took 68 seconds. On a 10,000-line module, it takes hours.

Do not use this when the tests are already known to be weak. Run coverage first. If coverage is below 80%, mutation testing will just confirm the obvious.

Do not use this as a merge gate without a human reviewing the survivors. Automated gates that block on surviving mutants produce equivalent-mutant fatigue, and developers start bypassing them.

The pattern

The workflow that keeps paying off: generate with a free model, verify with a cheap oracle, and measure the verification itself. The free model's tests were a starting point, not a conclusion. Mutation testing turned "all tests pass" into "14 specific behaviors are unverified."

That is the difference between a test suite that looks green and a test suite that earns its green. If you want to run this kind of experiment cheaply, MonkeyCode's free model access and free server option are a reasonable place to start — the harness above runs unchanged on it.