C++ Mutation Testing: Lift Weak Tests from 40% to 100%

Green tests and 100% line coverage can still miss real behavior. I ran C++ mutation testing on a free AI server: five mutants scored 40%, and two new tests took that score to 100%.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free tier. The free server ran the compile-and-test loop. The free tokens generated the mutants.

Why Green C++ Tests and 100% Coverage Still Fail

Green tests feel safe. They are not. They only prove what they already assert. Coverage metrics lie in a quieter way: they count lines, not behavior. A test that executes a line does not verify it. It just runs it.

Mutation testing makes that gap visible. You introduce one small bug, then rerun the existing suite. If the tests still pass, the mutant survived, which means the suite missed that behavior. I compute mutation score as killed divided by killed plus survived. Mutants that fail to compile do not count as kills.

Compare the two questions side by side:

• Coverage asks whether a line ran.
• Mutation asks whether a bug on that line would fail a test.
• Equivalent or nonsense mutants need a human to discard, or the score inflates.

I kept the experiment small on purpose. One function. Five mutants. A scorecard I could read without a cluster. Compiler-level C++ tools such as Mull exist for larger trees; I wanted a free, one-function loop first so I could inspect every survivor by hand.

A 100% Covered Function with Three Weak Tests

I picked vowel counting. Interview-scale. Easy to mutate.

int count_vowels(const std::string& s) {
    int count = 0;
    for (char c : s) {
        if (c == 'a' || c == 'e' || c == 'i' || c == 'o' || c == 'u') {
            count++;
        }
    }
    return count;
}

Three tests. Empty string. Consonants. Mixed case. All green. Coverage 100%.

TEST(VowelCount, HandlesEmpty) {
    EXPECT_EQ(count_vowels(""), 0);
}
TEST(VowelCount, HandlesConsonants) {
    EXPECT_EQ(count_vowels("bcdfg"), 0);
}
TEST(VowelCount, HandlesMixed) {
    EXPECT_EQ(count_vowels("Hello World"), 3);
}

Gaps I left, then forgot I had left:

• No string of all vowels ("aeiou")
• No input that starts with a vowel
• No uppercase vowels
• No input that even contains 'a' ("Hello World" has none)

Those omissions are the whole story. The tests confirm the obvious path. They never challenge a single letter, a single index, or case. That is the kind of suite mutation testing is designed to embarrass.

Generate, Filter, and Run Five C++ Mutants

I asked MonkeyCode's free model for mutants. One prompt. Five mutations.

Here is a C++ function. Generate 5 mutated versions.
Each mutation must introduce exactly one small bug.
Examples: change an operator, swap a constant, remove a condition.
Return only the mutated function bodies.

The model returned five variants. Two were useful. Three were nonsense: one changed the signature, one added a dependency, one returned a different type. Lesson one: AI mutants need human filtering. I kept only single-edit bodies that still compiled as int count_vowels(const std::string&).

My filter rules were simple:

• Exactly one behavioral change
• Same function signature
• No new includes or types
• Must compile with g++ -std=c++20

Each filtered mutant went through the same pipeline.

#!/usr/bin/env bash
set -euo pipefail

for mutant in mutants/*.cpp; do
    g++ -std=c++20 -c "$mutant" -o /tmp/mutant.o 2>/dev/null || continue
    g++ -std=c++20 /tmp/mutant.o tests/test_main.cpp -o /tmp/mutant_test
    if /tmp/mutant_test > /dev/null 2>&1; then
        echo "SURVIVED: $mutant"
    else
        echo "KILLED: $mutant"
    fi
done

What I actually did, in order:

1. Save each filtered mutant under mutants/ with the original signature.
2. Compile the mutant. Skip it if compilation fails.
3. Link against the existing test file.
4. Run the binary. Exit zero is SURVIVED. Non-zero is KILLED.
5. Score only mutants that compiled and linked.

Five mutants. Five compiles. Five test runs. The free server finished in under a minute. No local setup on my laptop. No CI bill. Compute was not the bottleneck. Review was.

How a 40% Score Named Two Missing Tests

Mutant	Mutation	Result
1	c == 'a' → c != 'a'	KILLED
2	count++ → count--	KILLED
3	c == 'a' → c == 'a' && false	SURVIVED
4	c == 'i' → c == 'I'	SURVIVED
5	loop starts at index 1	SURVIVED

Mutation score: 40%. Two of five killed. Three survived.

Why three mutants survived

• Mutant 3 dropped 'a' from the vowel set. No test string contained 'a'.
• Mutant 4 looked for 'I' instead of 'i'. No test used 'i' or 'I'.
• Mutant 5 skipped the first character. No test started with a vowel. "Hello World" starts with H. "bcdfg" starts with b.

The root cause was the same. I wrote tests to confirm the obvious. I did not write tests that would fail if one letter, one index, or one case changed.

The two tests that moved the score to 100%

The scorecard told me what to add:

TEST(VowelCount, HandlesAllVowels) {
    EXPECT_EQ(count_vowels("aeiou"), 5);
}
TEST(VowelCount, HandlesUppercaseVowels) {
    EXPECT_EQ(count_vowels("AEIOU"), 5);
}

HandlesAllVowels is the killer on this set. It contains 'a', contains 'i', and starts with a vowel, so mutants 3, 4, and 5 all fail it. After those tests, the score on this five-mutant card jumped from 40% to 100%. That is the power of mutation testing: it names the missing assertions instead of asking you to guess.

I still treat HandlesUppercaseVowels as a specification check. The production function only compares lowercase literals, so that test will fail until the function is case-insensitive or the expected value is 0. Either decision is stronger than leaving case untested.

Before versus after on this card:

• Before: three happy-path tests, 100% coverage, 40% mutation score
• After: five tests, same production function, 100% mutation score on these five mutants
• Cost: two assertions, not a rewrite of the suite

When a Free AI Server Is Enough—and When to Skip

Mutation testing is expensive on large codebases. You eventually want a dedicated engine and a cluster. For one function, a free tier is enough: five mutants, five compiles, five runs, under a minute. The free server did not break a sweat. JVM teams often start with PIT; the same idea applies here, just at interview-function scale.

The real work is human:

• Filter nonsense mutants.
• Decide which survivors matter.
• Add the smallest tests that kill them.
• Re-run the loop.

What this does not prove:

• The numbers are not universal. One function. Five mutants. One model.
• AI mutants are noisy. Three raw variants were unusable.
• A high score means your tests catch the mutations you tried. It says nothing about mutations you did not try.
• Mutation testing does not prove correctness. It proves test strength.

Skip this workflow if you have no tests, if your score is already high on mutants you care about, or if you cannot review mutants. Blind trust defeats the purpose.

Coverage is not strength. Tests can be green and weak at the same time. Free AI tokens are enough for a small C++ scorecard. The free server runs the loop. You bring the tests and the judgment.

Pick one 20-line C++ helper this week. Reuse three tests. Generate five single-edit mutants. Filter the nonsense. Compute killed / (killed + survived). If the score is under 80%, add the tests the survivors demand and re-run. MonkeyCode's free tier is enough for that one-function experiment. Mutate it. Score it. Find the holes before your users do. Then write down your before-and-after score so the next function starts from evidence, not from a green badge.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.