DEV Community

Cover image for Mutation testing in 5 minutes: catch the tests that secretly test nothing
Krishnasai Rollamadugu
Krishnasai Rollamadugu

Posted on

Mutation testing in 5 minutes: catch the tests that secretly test nothing

#testing #python #qa #tutorial

Your test suite is green. 100% passing. Maybe even 90% coverage.

Here's the uncomfortable question: if someone introduced a real bug right now, would your tests actually go red?

Coverage can't answer that — it only tells you a line ran, not that anything checked the result. A test can execute every line and assert nothing meaningful. It'll be green forever, including when the code is broken.

The technique that does answer it is mutation testing, and you can try it on your own code in about five minutes.

The idea in one sentence

Change the code to introduce a bug on purpose, run your tests, and see if they catch it.

Each small deliberate change is a "mutant" — flip a + to a -, a > to a <, a True to a False. For every mutant you run your suite:

  • tests fail → the mutant is killed → good, your tests caught the bug.
  • tests still pass → the mutant survived → a blind spot. Your tests didn't notice the code changed behaviour.

Your mutation score = killed / total. It's an honest measure of fault-detection, not just execution.

Try it in 5 minutes

I'll use a tiny open-source tool I built, trust-gate — zero dependencies, works with any test runner. (If you're on the JVM or JS, use PIT or Stryker instead — same idea.)

git clone https://github.com/rollamadugu/trust-gate
cd trust-gate
pip install -e ".[dev]"
Enter fullscreen mode Exit fullscreen mode

The repo ships a tiny pricing.py with a weak test suite and a strong one. Both are 100% green. Watch what mutation testing says about each.

Weak suite — it runs the code but barely checks the values:

trust-gate examples/pricing.py -c "pytest examples/test_pricing_weak.py -q" --cwd examples
Enter fullscreen mode Exit fullscreen mode 
mutants         : 7
caught (killed) : 0
survived        : 7
mutation score  : 0.00%
gate: FAIL
Enter fullscreen mode Exit fullscreen mode

Zero. Every injected bug slipped through while the suite stayed green.

Strong suite — same code, same "all passing", but it checks real values across branches:

trust-gate examples/pricing.py -c "pytest examples/test_pricing_strong.py -q" --cwd examples
Enter fullscreen mode Exit fullscreen mode 
mutants         : 7
caught (killed) : 7
survived        : 0
mutation score  : 100.00%
gate: PASS
Enter fullscreen mode Exit fullscreen mode

Same code, both green, only one would catch a bug

Same green checkmark on your dashboard. Completely different safety nets.

Reading the result

The useful part isn't the score — it's the surviving mutants. trust-gate prints each one with a line number and what it changed:

SURVIVED  line 9: Gt -> Lt
SURVIVED  line 11: 2 -> 3
Enter fullscreen mode Exit fullscreen mode

Each survivor is a precise to-do: "nothing you wrote would notice if line 9's comparison flipped." That points you straight at the assertion you're missing — far more actionable than a coverage percentage.

Make it a CI gate

Because trust-gate exits non-zero when the score is below a threshold, you can fail the build when test strength drops:

- run: pip install trust-gate
- run: trust-gate src/your_module.py -c "pytest -q" --min-score 0.75
Enter fullscreen mode Exit fullscreen mode

Now "green" means "and the tests would actually catch a bug," not just "ran without error."

Caveats (so you use it well)

  • Equivalent mutants: some changes don't alter behaviour, so they can't be killed. Don't chase 100%; chase understanding your survivors.
  • Speed: mutation testing re-runs your suite many times. Start on one important module, not the whole repo.
  • It grades the tests, not the code. A surviving mutant means your tests have a gap — exactly what you want to know.

The takeaway

Coverage tells you what ran. Mutation testing tells you what your tests would actually catch. Point it at your most important module today and look at the survivors — it's usually a humbling five minutes, in the best way.

Tool used: trust-gate (MIT, open source). Feedback and PRs welcome.

GitHub logo rollamadugu / trust-gate

Mutation-based test-strength gate: fail CI when your passing tests are too weak to catch real bugs. Zero dependencies, any test runner.

trust-gate

Python Dependencies License

Your tests are green. But are they strong enough to catch a real bug?

trust-gate injects small faults (mutants) into your code, runs your test command for each one, and measures how many faults your passing tests actually catch. Green tests that miss injected faults are weak tests — trust-gate makes that measurable and lets you fail CI when test strength drops below a threshold.

It's a focused, zero-dependency mutation-testing gate. Pure Python standard library. Works with any test runner (pytest, unittest, a shell script — anything that exits non-zero on failure).

Part of a broader line of work on trust in AI-driven testing (TrustLab). As AI writes more of our tests, "it passed" needs to mean "it would have caught the bug." This gate checks exactly that.

Install

pip install trust-gate        # from PyPI (coming soon)
# or, from source:
pip install -e ".[dev]"
Enter fullscreen mode Exit fullscreen mode
…
View on GitHub

I'm Krishnasai Rollamadugu — a test automation architect researching trust and measurement in AI-driven software testing. If weak-but-green tests are your kind of problem, I write about this a lot.

Top comments (0)