Characterization Tests Can Be Fake. Mutation Testing Proves They Work.

A passing characterization test can still be useless.
Mutation testing reveals which tests can actually catch a behavior change.
Use it before you refactor. Then make the smallest safe change.

The False Oracle

Characterization tests lock in current behavior.
They pass when behavior stays the same.
But many of them pass even when behavior changes.

Why?
Because they often assert only a return value.
They ignore side effects, branches, and edge cases.
You refactor, tests stay green, and production breaks.

That is the false oracle problem.

Mutation Testing Fixes That

Mutation testing makes small changes to your code.
It flips a plus to a minus, a true to a false.
Then it runs your tests.

If a test still passes after the mutation, the mutant survived.
A surviving mutant shows a gap in your test suite.

You want mutants to die.
Surviving mutants mean your refactor safety net has holes.

A Tiny Python Example

Here is a legacy function:

# legacy.py
def apply_discount(price, rate):
    return price * rate

A simple characterization test might look like this:

# test_legacy.py

def test_apply_discount():
    assert apply_discount(100, 0.9) == 90

That test verifies one happy path.
It does not verify zero, negative, or float precision.
It also does not verify what happens when rate is larger than 1.

Now run mutation testing with mutmut:

pip install mutmut
mutmut run --paths-to-mutate legacy.py
mutmut results

mutmut will mutate * to /, among other changes.
The test above catches that mutant for the case 100 * 0.9.
But a mutant in an untested branch will survive.

That result tells you exactly what you still do not know.

Step 1: Generate Edge Cases Quickly

Manually thinking of edge cases can take minutes.
I used MonkeyCode's free model to suggest a list of boundary inputs.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.

An example prompt could be:

Given this function, list five test inputs that include zero, negative, floating point, and extreme values.

The model returned useful candidates.
I then turned those into parametrized characterization tests.

Running those tests on MonkeyCode's free server cost nothing extra.
That kept the whole experiment close to zero budget.

Step 2: Measure Your Mutation Score

Run the mutation suite again after adding the edge cases.
Check the score:

mutmut results

A score below 60% means your tests only guard happy paths.
A score above 70% is a reasonable floor for a refactor target.

Do not chase 100%.
Some mutants are equivalent code changes.
They produce logically identical programs and will survive forever.

Use the report as a map.
Focus on the functions you plan to modify.

Step 3: Make the Smallest Safe Change

Now that your tests are stronger, pick one behavior-preserving change.
Change one line, one variable, or one branch at a time.

After each change:

• Run the unit tests.
• Run the mutation suite again.
• Check that the mutation score does not drop.

If the score drops, you lost test coverage.
If a previously killed mutant now survives, your change hid a behavior.

Revert and try a smaller slice.

When to Skip This Workflow

Mutation testing is not always the right tool.

Skip it if:

• Your test suite takes over five minutes.
• You are only renaming a local variable.
• The module is already under strong property-based tests.

Mutation testing is slow on large suites.
It can generate thousands of mutants for one modest file.

It also generates false confidence if you ignore equivalent mutants.

Use it selectively.
Apply it to the exact function you plan to refactor, not the whole repo.

Limitations and Tooling

This example used Python and mutmut.
Other languages have similar tools:

• JavaScript / TypeScript: Stryker
• Java: PIT
• Go: go-mutesting

Each tool has its own syntax and config.
The principle is the same: break the code, see if tests fail.

Mutation testing is not a substitute for code review.
It is a measurement of test strength.

Final Thought

A characterization test is only useful if it hurts when change happens.
Mutation testing shows you where the pain is missing.

Write good tests.
Measure them with mutants.
Then make the smallest refactor you can defend.

That is how you refactor a messy repo without lying to yourself.