I Wrote a Tool to Find Blind Spots. It Had One.

I spend a lot of time writing quality gates to enforce invariants in my software. For weeks, my test runner stayed green. Every guard was in place, every check passed, and every test suite reported clean execution.

Then I asked an uncomfortable question: are these gates actually measuring anything, or are they just executing without complaining?

To find out, I wrote a script to sabotage my own engine. The tool parses the source code, finds every fail() assertion call in the engine, disables them one by one, and re-runs the entire mutation test suite for each line. It asks a simple question: if I delete this guard line, does any test notice?

If no test notices, that line is uncovered. It isn't a protection; it's a comment pretending to be a guard.

I expected to find a few gaps. I even wrote that expectation directly into the script's exit codes, annotating exit code 1 as (expected in v2.4.1). I knew I had blind spots—I just didn't know how many.

When the measurement finished across 61 protection lines in the engine, 40 of them were completely unmeasured by any test—a 66% blind spot rate.

Yet, the baseline test suite run right beside it reported: 36/36 biting mutants. A 100% green test suite, sitting on top of an engine where two-thirds of the assertions could be erased without triggering a single alert.

0  her fail() kapsamli (hicbir kor nokta yok)
1  en az bir KAPSAMSIZ fail() var  (v2.4.1'de beklenen)
2  OLCULEMEDI / kurulum hatasi

# translation:
# 0  every fail() is covered (no blind spots)
# 1  at least one UNCOVERED fail() exists (expected in v2.4.1)
# 2  UNMEASURABLE / setup error

Three ways a gate dies

A quality gate usually dies in one of three ways. All three look identical from the outside: they give you a green checkmark.

• The Dead Gate. It fell out of the test runner or CI pipeline. The script file sits in the repository, but nothing invokes it anymore. It never fails because it never runs.
• The Environmentally Broken Gate. It gets called by CI, but it cannot run properly on that specific platform due to missing dependencies, OS permission boundaries, or character encoding mismatches. It returns green because its internal errors are caught under a soft error flag.
• The Blind Gate. It runs. It finishes with exit code 0. And it measures absolutely nothing because the input data required to trigger its logic is never produced.

Most engineering setups have answers for the first two: linters, pipeline validation, and coverage runners. Almost nobody writes tests for the third.

Handling environmental failure requires thinking about what happens when the measurement tool itself gets corrupted by the environment. For instance, in my repository, the hukum_kapisi.py gate uses explicit ASCII character patterns for string matching rather than UTF-8 dashes. Why? Because when an encoding error triggers Python's errors="replace" handler, a hyphen or dash can be printed as a ?. If the gate matched against unicode dashes, the very encoding failure it was designed to catch would corrupt the pattern matcher itself.

There is a vast difference between "I wrote a gate" and "I thought about what happens to the gate when its environment breaks."

Level 1: mutate the gate — and why it isn't enough

The standard response to testing quality gates is Level 1 mutation: intentionally break the gate's code, run it against dirty input, and confirm that exit != 0.

This is a good reflex, but it is incomplete. A gate requires two-way proof:

• It must bite on dirty input.
• It must stay silent on clean input.

If you only test that a gate triggers on bad input, you risk keeping a gate that constantly fires false positives on clean environments.

More importantly, proving that a gate bites on a synthetic input does not prove that it covers the entire class of errors it claims to guard. Having a single mutant test pass against a gate gives you a false sense of security.

Level 2: mutate the engine, ask if any gate notices

Level 2 flips the approach: don't mutate the gate. Mutate the engine that the gates are guarding, then check if any gate notices the change.

My script, sabotaj.py, automates this process through three specific design rules:

• AST parsing, never regex. Searching for function calls in source code using regular expressions inevitably misses multi-line calls. When a measurement tool misses a line silently, the tool itself becomes blind. An analysis script must parse the actual grammar of the language.
• Read-only execution copies. The script never touches the working source file hafiza.py. Each sabotage operation creates a temporary memory/disk copy, runs the suite, and discards the copy. If a tool modifies its own source of truth during execution, it is no longer measuring—it is mutating.
• The compile() syntax gate. If a sabotaged file fails Python's compile() step, the result is classified as UNMEASURABLE, not UNCOVERED. Conflating broken syntax with missing test coverage artificially inflates blind spot metrics.

# 1) fail() cagri yerlerini AST ile bul  (regex DEGIL: cok satirli cagrilar var)
# translation: 1) Find fail() call sites via AST (NOT regex: multi-line calls exist)
def fail_cagrilari(kaynak):
    agac = ast.parse(kaynak)
    bulunan = []
    for d in ast.walk(agac):
        if isinstance(d, ast.Call) and isinstance(d.func, ast.Name) and d.func.id == "fail":
            ...

def sabote_et(kaynak, hedef):
    """Tek bir fail(...) cagrisini `None` ile degistir. Kaynagi DEGISTIRMEZ."""
    # translation: Replace a single fail(...) call with `None`. Does NOT modify source.
    ...
    compile(yeni, "<sabotaj>", "exec")   # sozdizimi bozulduysa burada patlar
    # translation: # explodes here if syntax is broken
    return yeni

When I ran this analysis against hafiza.py, the core value was not the total number of 40 unmeasured lines—it was how those blind spots clustered. Looking at key selected gates from the results:

Gate Identifier	Comprehensive (Covered)	Uncovered (Blind)
H1	0	6
H9	0	1
H11	1	10
H8	2	4
H3 / H7 / H15	1 / 1 / 2	0

Gate H11 is the clearest example. Out of 11 fail() protection calls inside H11, 10 were completely unmeasured. Yet, because 1 mutant (M-H11) triggered a single line, the gate marked itself as "tested" in naive summary reports. Testing a single sink line with one mutant does not cover the gate's invariant class.

The third verdict — and the day my own tool lost it

Every measurement tool must support a third verdict: UNMEASURABLE (ÖLÇÜLEMEDİ).

Binary PASS/FAIL models force tools to make false statements when execution environment issues prevent proper testing. I learned this when an earlier CI run hit a read-only filesystem restriction, threw a raw PermissionError, and returned exit code 1. The contract declared 1 = MUTANT ESCAPED. The runner had reported an unmeasurable state as if it were a legitimate test failure. Fake failures destroy trust in real failures.

While reading through sabotaj.py line by line to prepare this writeup, I looked at how main() evaluates its final exit code:

if olculemedi and not kapsamsiz:
    print("HUKUM: OLCULEMEDI kalemler var — 'tam kapsamli' DEMEK YASAK.")
    return 2
if kapsamsiz:
    print("HUKUM: %d KOR NOKTA var." % len(kapsamsiz))
    return 1

# translation:
# if unmeasured and not uncovered:
#     print("VERDICT: UNMEASURABLE items exist — FORBIDDEN to claim 'fully covered'.")
#     return 2
# if uncovered:
#     print("VERDICT: %d BLIND SPOTS exist." % len(uncovered))
#     return 1

Look at the logical condition in that first branch: if olculemedi and not kapsamsiz:.

If the script encountered both unmeasurable compilation errors AND uncovered lines during a run, not kapsamsiz evaluated to False. The code bypassed the first block entirely and dropped straight into if kapsamsiz:, returning exit code 1. The warning explicitly forbidding a claim of full coverage would never execute, and exit code 2 was swallowed.

The third verdict existed in a print statement, but it had dropped out of the exit code contract.

In my actual measurement run, ÖLÇÜLEMEDİ happened to be 0, so the script reported its final count correctly. I didn't find this bug because a test failed or an alert fired—I found it because I read the source code line by line.

It was a latent defect sitting silently inside the script's main() function. The tool I wrote to find blind spots had a blind spot in its contract logic.

What this does NOT tell you

To maintain clear boundaries around what these metrics mean, three constraints must be stated directly:

• The count of 40 is an upper bound. sabotaj.py parses output logs from the test runner using regex:

  kacanlar = sorted(set(re.findall(r"(M-[A-Z0-9a-z_]+)\s+.*?KACTI", cikti)))
  if not kacanlar:
      kacanlar = sorted(set(m for m in re.findall(r"^\s*(M-\S+).*KACTI", cikti, re.M)))

If both regexes fail to match an escaped mutant name in the log output, kacan evaluates to 0, which defaults the verdict to KAPSAMSIZ (uncovered). The tool that uses strict AST parsing for input files relies on fragile string parsing for output reading.

• Docstrings decay faster than code. The script docstring declared (bugun 60 adet) (60 items today), but the AST parser returned 61 active fail() calls. Docstrings are historical statements that are accurate only on the day they are written.
• Coverage is not correctness. A line marked as COMPREHENSIVE simply means that disabling it caused a test to fail. It does not prove that the guard is evaluating the correct business invariant or handling edge cases properly.

Try it on your repo

If you want to verify your own quality gates, the approach can be implemented in any codebase using a simple sequence:

• Parse your engine's abstract syntax tree (AST) to locate every assertion or failure guard. Do not use regular expressions for code structure detection.
• Iterate through each call site, replacing it with a no-op (None or pass) inside an isolated temporary file copy.
• Run compile() on the modified source to catch syntax breaks before testing.
• Execute your test suite against each single-line modification.
• If the test suite stays green, log that line location as uncovered.
• Ensure your runner's exit logic isolates execution errors (UNMEASURABLE) from assertion failures (UNCOVERED) without conditional short-circuits.

For every fix you apply to a system, you owe it a sabotage: strip the fix out temporarily, rerun the suite, and verify that the exact target error reproduces. If removing the fix doesn't bring back the original failure, your test is measuring a neighboring component, not the bug.

Measured at commit a86ffe4 (a86ffe440c0740241fdf20387c14420029ec0555), 1 Aug 2026.

To verify locally:

git clone https://github.com/onur-kesim/hafiza-kur.git
cd hafiza-kur
git checkout a86ffe4
python faz0/sabotaj.py --motor skill/scripts/hafiza.py

# The run above prints the counts. To see the defect itself — the branch that
# swallows the third verdict — read the contract logic directly:
grep -n -A 4 "if olculemedi and not kapsamsiz" faz0/sabotaj.py

Comments

[Vinh Nguyen]

The three verdicts are easier to keep honest if aggregation is a precedence table rather than branch order: UNMEASURABLE should dominate UNCOVERED, which should dominate PASS. A table-driven test for the four (unmeasurable, uncovered) combinations would have caught the swallowed mixed state without needing another mutation.

[Onur Kesim]

You're right, and I checked before answering rather than after: the branch is still there. On main today, faz0/sabotaj.py still reads if olculemedi and not kapsamsiz: — the grep command in the post still reproduces. Last commit touching that file was 18 August. So this is not "fixed, thanks for the note"; it's an open defect I'm reporting the state of.

The part of your comment I'd underline is the difference between our two instincts. Mine was to reorder the branches. Yours removes ordering from the decision entirely. Branch order is a precedence table — an unnamed one, with no place to write the expected values down, which is exactly why nobody tested it. You cannot table-drive a table you never wrote.

Written out, the four combinations are: neither flag set is PASS; uncovered alone is UNCOVERED; unmeasurable alone is UNMEASURABLE; and both set is UNMEASURABLE.

The bug is the last combination only. The tool returned UNCOVERED there, which is the more reassuring of the two — it says "I looked and found blind spots" instead of "I could not look." That direction is not an accident; a tool that degrades toward the more confident answer is the failure mode worth naming.

One caveat I'd add to your own proposal, against my interest: a table-driven test catches this only if the expected column is written from the invariant. If I had generated it by running the current function and pasting the output, the test would have frozen the bug and then defended it. For a tool whose whole job is certifying other tools, that is not a hypothetical risk — it is the same mistake one level up.

[Kane Lim]

This is a very important distinction: a green test suite proves execution, not necessarily measurement.

The Level 2 approach is especially interesting because it changes the question from “does this test fail when I inject a bad case?” to “what happens when I remove the exact protection mechanism the system relies on?” That gives you much stronger evidence that the test suite is actually coupled to the invariant being protected.

I also like the explicit UNMEASURABLE state. In production engineering, treating infrastructure/tooling failures as ordinary PASS/FAIL results is dangerous because it creates false confidence. A measurement system needs to be able to say “I don't know” rather than silently converting uncertainty into success.

The olculemedi and not kapsamsiz issue is a great example of why the measurement framework itself needs testing. There is an interesting recursive principle here:

If a system is responsible for proving that something is protected, the proof mechanism itself must be subject to adversarial testing.

One additional direction I'd explore is combining this with property-based testing and differential mutation selection. Rather than mutating every fail() independently, you could prioritize mutations by invariant, execution path, or historical defect density. That could make large repositories substantially cheaper to analyze while preserving the useful signal.

The “every fix owes a sabotage” idea is also compelling. It turns regression tests into evidence that the specific protection mechanism—not merely some neighboring behavior—is actually necessary.

This is the kind of testing infrastructure that becomes increasingly valuable as systems and automated code generation grow more complex. I'd be interested in comparing approaches and discussing how to make this practical across larger Python codebases.

[Onur Kesim]

Thank you — the recursive line ("if a system is responsible for proving that something is protected, the proof mechanism itself must be subject to adversarial testing") is a cleaner statement of the thesis than the one in the post.

On prioritizing mutations by invariant, execution path, or historical defect density: I have no measurement, so I won't pretend to an opinion. The run in the post is small enough that cost isn't binding yet, which means I have no evidence about whether prioritization preserves the signal or quietly discards the residual — and given what the post is about, "quietly discards the residual" is the outcome I'd have to rule out before recommending it, not after.

Worth noting for anyone reading this thread: the specific defect discussed here is still present on main. The grep command in the post still reproduces it.