Your test suite is green. 100% passing. Maybe even 90% coverage.
Here's the uncomfortable question: if someone introduced a real bug right now, would your tests actually go red?
Coverage can't answer that — it only tells you a line ran, not that anything checked the result. A test can execute every line and assert nothing meaningful. It'll be green forever, including when the code is broken.
The technique that does answer it is mutation testing, and you can try it on your own code in about five minutes.
The idea in one sentence
Change the code to introduce a bug on purpose, run your tests, and see if they catch it.
Each small deliberate change is a "mutant" — flip a + to a -, a > to a <, a True to a False. For every mutant you run your suite:
- tests fail → the mutant is killed → good, your tests caught the bug.
- tests still pass → the mutant survived → a blind spot. Your tests didn't notice the code changed behaviour.
Your mutation score = killed / total. It's an honest measure of fault-detection, not just execution.
Try it in 5 minutes
I'll use a tiny open-source tool I built, trust-gate — zero dependencies, works with any test runner. (If you're on the JVM or JS, use PIT or Stryker instead — same idea.)
git clone https://github.com/rollamadugu/trust-gate
cd trust-gate
pip install -e ".[dev]"
The repo ships a tiny pricing.py with a weak test suite and a strong one. Both are 100% green. Watch what mutation testing says about each.
Weak suite — it runs the code but barely checks the values:
trust-gate examples/pricing.py -c "pytest examples/test_pricing_weak.py -q" --cwd examples
mutants : 7
caught (killed) : 0
survived : 7
mutation score : 0.00%
gate: FAIL
Zero. Every injected bug slipped through while the suite stayed green.
Strong suite — same code, same "all passing", but it checks real values across branches:
trust-gate examples/pricing.py -c "pytest examples/test_pricing_strong.py -q" --cwd examples
mutants : 7
caught (killed) : 7
survived : 0
mutation score : 100.00%
gate: PASS
Same green checkmark on your dashboard. Completely different safety nets.
Reading the result
The useful part isn't the score — it's the surviving mutants. trust-gate prints each one with a line number and what it changed:
SURVIVED line 9: Gt -> Lt
SURVIVED line 11: 2 -> 3
Each survivor is a precise to-do: "nothing you wrote would notice if line 9's comparison flipped." That points you straight at the assertion you're missing — far more actionable than a coverage percentage.
Make it a CI gate
Because trust-gate exits non-zero when the score is below a threshold, you can fail the build when test strength drops:
- run: pip install trust-gate
- run: trust-gate src/your_module.py -c "pytest -q" --min-score 0.75
Now "green" means "and the tests would actually catch a bug," not just "ran without error."
Caveats (so you use it well)
- Equivalent mutants: some changes don't alter behaviour, so they can't be killed. Don't chase 100%; chase understanding your survivors.
- Speed: mutation testing re-runs your suite many times. Start on one important module, not the whole repo.
- It grades the tests, not the code. A surviving mutant means your tests have a gap — exactly what you want to know.
The takeaway
Coverage tells you what ran. Mutation testing tells you what your tests would actually catch. Point it at your most important module today and look at the survivors — it's usually a humbling five minutes, in the best way.
Tool used: trust-gate (MIT, open source). Feedback and PRs welcome.
rollamadugu
/
trust-gate
Mutation-based test-strength gate: fail CI when your passing tests are too weak to catch real bugs. Zero dependencies, any test runner.
trust-gate
Your tests are green. But are they strong enough to catch a real bug?
trust-gate injects small faults (mutants) into your code, runs your test
command for each one, and measures how many faults your passing tests
actually catch. Green tests that miss injected faults are weak tests —
trust-gate makes that measurable and lets you fail CI when test strength
drops below a threshold.
It's a focused, zero-dependency mutation-testing gate. Pure Python standard library. Works with any test runner (pytest, unittest, a shell script — anything that exits non-zero on failure).
Part of a broader line of work on trust in AI-driven testing (TrustLab). As AI writes more of our tests, "it passed" needs to mean "it would have caught the bug." This gate checks exactly that.
Install
pip install trust-gate # from PyPI (coming soon)
# or, from source:
pip install -e ".[dev]"
I'm Krishnasai Rollamadugu — a test automation architect researching trust and measurement in AI-driven software testing. If weak-but-green tests are your kind of problem, I write about this a lot.

Top comments (4)
Mutation score is the first common metric that grades the suite instead of the code, which is why it needs the same warning label as every other ratio: it hides rows.
I hit this in a different domain. Tokenizing 10,000 real prompts, the batch total came out 5% off while 40.5% of individual prompts were off by more than 20%, spanning -24% to +47%. The aggregate earned a passing grade. Four in ten rows were still wrong by a fifth.
A mutation score of 80% can be 100% on the auth module and 20% on the billing module, and nobody finds out if the dashboard prints one number. So the column I'd add next to killed / total is where the survivors live: survive rate per module, not per suite. Score tells you the suite is alive; the per-module column tells you which part is dead.
It's also the automated version of a rule I argued on a security audit thread here: a check that passes forever decays into theater. Mutants are that bite, on a schedule, without needing someone to volunteer for it.
@devenquan ,This is the better framing, thanks. Mutation score grades the suite, so printing one number for the whole repo buries the thing you actually want to know. Your auth-100 / billing-20 case is the failure mode exactly: a global 80% looks healthy while billing is effectively untested, and nobody reads the suite as "one module is dead."
I'd push the per-module column one step further, to the survivor list. The rate tells you which module is dead; the list of surviving mutants tells you the specific assertion that's missing. Each survivor is a line you can point at and say "nothing tested this." So the score means the suite is alive, survive-rate-per-module means which part is dead, and the survivor list is what to write next. In CI I gate on new survivors introduced by a diff rather than a global threshold, for the same reason you'd never let the billing column hide behind auth.
And yes to the last point. A check that can't fail is theater, and mutation testing is the scheduled, no-volunteer-needed version of that audit rule: a green assertion-free test passes forever, and a mutant is the thing that bites it without anyone remembering to look. Your tokenizer example lands the same way, 5% in aggregate but 40% of rows wrong by a fifth. Same lie, different ratio.
The survivor list is the right third step — per-module rate tells you where to look, the list tells you what to write. Taking it. One hole in gating on survivors introduced by a diff: a diff only touches files that changed, so a new survivor can only ever be born inside a module someone edited. A module nobody has opened in eighteen months produces zero new survivors and keeps every old one, which makes its rot invisible to a diff-scoped gate. The gate ends up measuring churn, not decay. That's why the list wants an age column next to it. "Nothing tested this" and "nothing has tested this since the last person who understood it left" are different tickets, and the second one is the one that bites.
One thing worth adding about reading survivors: relational-operator mutants are a different signal from the rest. When Gt > Lt survives, it usually doesn’t mean you need more assertions, it means you have no case near the boundary at all, so every input you test sits comfortably on one side of the comparison. The fix is a test design fix, not an assertion fix.
That also makes mutation score a decent sanity check on boundary coverage, which is otherwise hard to measure.