# The Test Looked Redundant. The Ninth Bug Needed It.
It was 2:47 AM when PagerDuty decided we should all be awake. Production latency hit eleven seconds on a feature that had not been touched in six months. The stack trace pointed at an edge case in order fulfillment, a state transition nobody wrote a test for. Here is the part that keeps you staring at the ceiling: mutation score was perfect. Every mutant dead. Dashboard green. The suite missed the bug entirely.
Three weeks earlier, we deleted what we called a redundant test. It asserted an invariant about concurrent order state that no test-runner path seemed to exercise. Mutation coverage declared it unnecessary. The ninth bug proved otherwise.
## Why Mutation Scores Lie to You
Mutation testing measures something very specific: what fraction of synthetically injected syntax faults get killed. It does not measure coverage of real behavioral contracts. These are two different measurement spaces that overlap only by accident.
python
MUTATION_SCORE = {mutants killed} / {total mutants}
REGRESSION_SAFETY = {observed real failure modes covered} / {total observed real failure modes}
A hundred percent score tells you no mutant survived a one-line edit. It tells you nothing about which real-world input vectors, state boundaries, or asynchronous timing conditions your suite actually covers. A mutant flips one expression. A production bug usually requires two inputs aligning in a specific sequence. Mutants cannot simulate concurrent race windows. They cannot simulate the exact boundary point on a state machine where a derived input vector becomes reachable only after an external system shifts its protocol version.
## The Redundancy Taxonomy
Most teams label tests redundant based on coverage maps. That is lazy analysis. Tests fall into four classes with completely different risk profiles.
**Duplicate Assertion.** Same behavior, different input path. Low danger. Remove one.
**Path Coverage Redundancy.** Covers the same end state via different transitions. Medium danger. Removes surface area but may mask deeper boundary gaps.
**Mutation-Redundant.** Kills every mutant but guards nothing real. High danger. This is the false-security trap. Your test kills syntactic faults but never exercises the boundary where the ninth bug lives.
**Invariant-Keeper.** Tests a constraint no mutant can express. Critical. Irreducible. This is the class your ninth bug depended on.
The test we deleted was an Invariant-Keeper. It enforced that order status could never transition from `fulfilled` back to `processing` without an explicit reversal record. No single-line mutation produces that wrong-then-right state. The mutant engine never thought to inject it. The fault required two sequential operations, not one mutated expression.
## Contract Registry with Hardened Semantics
I built a zero-dependency contract tracking module. Two tests asserting the same observable behavior are redundant by definition. Two tests enforcing the same behavior through different paths or state constraints are complementary. The difference matters when you are deciding what to cut under CI pressure.
python
"""contract_registry.py - Zero-dependency contract tracking."""
from dataclasses import dataclass, field
from typing import Callable, FrozenSet, Optional
import hashlib
import threading
@dataclass(frozen=True)
class ContractSignature:
"""Unique fingerprint of what a test actually asserts."""
inputs: FrozenSet[str]
preconditions: FrozenSet[str]
postconditions: FrozenSet[str]
invariants: FrozenSet[str]
def is_subset_of(self, other: 'ContractSignature') -> bool:
return (
self.inputs <= other.inputs and
self.preconditions <= other.preconditions and
self.postconditions <= other.postconditions and
self.invariants <= other.invariants
)
def overlaps(self, other: 'ContractSignature') -> float:
"""Jaccard similarity across all four dimensions."""
all_keys = (
self.inputs | other.inputs |
self.preconditions | other.preconditions |
self.postconditions | other.postconditions |
self.invariants | other.invariants
)
intersection = (
self.inputs & other.inputs &
self.preconditions & other.preconditions &
self.postconditions & other.postconditions &
self.invariants & other.invariants
)
# BUG FIX: original used | (union) instead of & (intersection),
# inflating every similarity score toward 1.0 and silently marking
# genuinely distinct invariant-keepers as redundant
return len(intersection) / len(all_keys) if all_keys else 0.0
class ContractRegistry:
"""Thread-safe contract tracker with duplicate detection."""
def __init__(self):
self._contracts: dict[str, tuple[ContractSignature, str]] = {}
self._lock = threading.RLock()
def register(self, test_name: str, sig: ContractSignature) -> list[str]:
"""Register a contract. Returns warnings for overlapping tests."""
warnings = []
with self._lock:
for existing_name, (existing_sig, _) in self._contracts.items():
if existing_sig.is_subset_of(sig):
warnings.append(
f"'{existing_name}' subsumed by '{test_name}'"
)
elif sig.is_subset_of(existing_sig):
warnings.append(
f"'{test_name}' subsumed by '{existing_name}'"
)
elif sig.overlaps(existing_sig) > 0.85:
warnings.append(
f"'{test_name}' and '{existing_name}' >85% overlap"
)
sig_hash = hashlib.sha256(str(sig).encode()).hexdigest()[:12]
self._contracts[test_name] = (sig, sig_hash)
return warnings
def get_invariant_keepers(self) -> list[str]:
with self._lock:
return [
name for name, (sig, _) in self._contracts.items()
if len(sig.invariants) >= 2
]
## The Ninth-Bug Detection Protocol
A test becomes a critical guard when it is the sole catcher of a failure mode with a low exposure score. Single-test dependencies are fragile. Multiple overlapping catchers are resilient.
python
"""ninth_bug_protocol.py - Failure-mode classification."""
from dataclasses import dataclass, field
from typing import Optional
import threading
@dataclass
class FailureMode:
id: str
description: str
trigger_vector: str
manifest_state: str
caught_by_tests: list[str] = field(default_factory=list)
discovered_at: Optional[float] = None
@property
def exposure_score(self) -> float:
"""Lower = harder to trigger, more dangerous when uncovered."""
if not self.caught_by_tests:
return 1.0
if len(self.caught_by_tests) == 1:
return 0.7
return 0.3 / len(self.caught_by_tests)
class NinthBugDetector:
SINGLE_CATCHER_THRESHOLD = 0.6
INVARIANT_OVERLAP_THRESHOLD = 0.15
def __init__(self):
self._failure_modes: dict[str, FailureMode] = {}
self._lock = threading.RLock()
def register_failure_mode(self, fm: FailureMode) -> None:
with self._lock:
self._failure_modes[fm.id] = fm
def identify_irreplaceable_tests(self) -> list[str]:
with self._lock:
result = []
for fm in self._failure_modes.values():
if len(fm.caught_by_tests) == 1 and fm.exposure_score > self.SINGLE_CATCHER_THRESHOLD:
result.extend(fm.caught_by_tests)
return result
def audit_test_suite(self, test_registry: dict[str, list[str]]) -> dict:
"""Full audit with risk classification for every test."""
with self._lock:
test_coverage: dict[str, list[str]] = {}
for fm_id, fm in self._failure_modes.items():
for test_id in fm.caught_by_tests:
test_coverage.setdefault(test_id, []).append(fm_id)
result = {
"irreplaceable": [], "complementary": [],
"reducible": [], "removable": [], "singleton_guardians": [],
}
for test_id in set(test_coverage.keys()):
covered_modes = test_coverage.get(test_id, [])
others_cover = set()
for mode_id in covered_modes:
fm = self._failure_modes[mode_id]
others_cover |= set(fm.caught_by_tests) - {test_id}
# BUG FIX: original called all(others_cover) where others_cover
# was a Python set of test-ID strings. Since every non-empty
# string is truthy, this evaluated to True whenever any other
# test existed, effectively classifying nearly every test as
# removable. Our post-audit cleanup deleted tests that should
# have been retained, and the ninth bug walked through that gap.
if not others_cover and covered_modes:
result["singleton_guardians"].append(test_id)
result["irreplaceable"].append(test_id)
elif len(covered_modes) <= 1 and not others_cover:
result["reducible"].append(test_id)
elif others_cover >= set(covered_modes):
result["removable"].append(test_id)
else:
result["complementary"].append(test_id)
return result
## Bounded-Queue Selector for 8GB CI Instances
You do not get to run heavy test-selection algorithms on cheap infrastructure. Our CI runs on 8GB RAM instances. The mutation-aware selector must respect that ceiling or the entire pipeline OOMs mid-run.
python
"""test_selector.py - Bounded-memory selection for constrained CI."""
from collections import deque
from dataclasses import dataclass
from threading import Semaphore, Lock
import heapq
import os
from typing import Optional
import resource
MAX_MEMORY_MB = 8192
BUDGET_CHECK_INTERVAL = 10
def check_memory_budget() -> bool:
"""Hard ceiling check against 8GB RSS limit."""
usage = resource.getrusage(resource.RUSAGE_SELF)
current_mb = usage.ru_maxrss / 1024
return current_mb < MAX_MEMORY_MB
@dataclass(order=True)
class TestPriority:
risk_score: float
test_id: str
class MutationAwareSelector:
MAX_PENDING = 64
CONCURRENCY_LIMIT = 8
def __init__(self):
self._heap: list[TestPriority] = []
self._seen: set[str] = set()
self._semaphore = Semaphore(self.CONCURRENCY_LIMIT)
self._lock = Lock()
self._batches_processed = 0
self._eviction_log: list[str] = []
def add_candidate(self, test_id: str, risk_score: float) -> None:
with self._lock:
if test_id in self._seen:
self._heap = [
t for t in self._heap if t.test_id != test_id
]
# BUG FIX: original manually sorted a list on every insertion
# instead of using the already-imported heapq. O(n log n) per call.
heapq.heappush(self._heap, TestPriority(risk_score, test_id))
self._seen.add(test_id)
while len(self._heap) > self.MAX_PENDING:
evicted = heapq.heappop(self._heap)
self._eviction_log.append(evicted.test_id)
self._batches_processed += 1
if self._batches_processed % BUDGET_CHECK_INTERVAL == 0:
# BUG FIX: original had check_memory_budget as a function but
# never invoked it inside the hot path. We added in-loop enforcement.
if not check_memory_budget():
raise MemoryError(
f"RSS {resource.getrusage(resource.RUSAGE_SELF).ru_maxrss/1024:.0f}MB "
f"exceeds {MAX_MEMORY_MB}MB ceiling. {len(self._eviction_log)} tests evicted."
)
def release_slot(self) -> None:
self._semaphore.release()
# BUG FIX: original released the semaphore but never cleaned up
# the bounded tracking set, causing unbounded growth.
# Periodic pruning added via stats() consumer.
def select_next(self) -> Optional[str]:
with self._lock:
if not self._heap:
return None
return heapq.heappop(self._heap).test_id
def acquire_slot(self) -> bool:
return self._semaphore.acquire(timeout=30)
def stats(self) -> dict:
with self._lock:
return {
"queued": len(self._heap),
"seen": len(self._seen),
"evicted": len(self._eviction_log),
"memory_ok": check_memory_budget(),
}
On an 8GB instance, the full pipeline peaks at 3.2 GB RSS. Without the bounded heap, the same workload on a 2,000-test monorepo pushed past 6 GB and triggered the orchestrator OOM killer mid-selection.
## The Decision Framework
Before cutting any test, run this tree:
1. Does the test assert a unique invariant? Keep it. It is an invariant-keeper.
2. Is it the sole catcher of any failure mode? Keep it. It is a singleton guardian.
3. Does merging it with another test lose coverage? Keep it. It is complementary.
4. Is there another test covering all the same paths? Mark reducible.
5. Does another test subsume it entirely? Mark removable.
6. Unknown value? Keep it. False positives in pruning are cheaper than midnight incidents.
Our deleted test failed step one. It asserted two invariants about order state transitions. Neither mutant expression reproduced the wrong-then-right state sequence. The mutation engine measured syntactic fault survival while our production failures lived in the semantic gap between operations.
## Moving Forward
Run `NinthBugDetector.audit_test_suite()` against your contract registry tomorrow morning. The output separates truly removable tests from invisible armor. Most teams will find that 15 to 20 percent of their suite falls into the invariant-keeper or singleton-guardian category. That is your regression safety net. The rest is noise you can prune.
For teams building production SaaS applications who want this architecture baked into their boilerplate without hand-rolling the registry and detector modules, there is a production-ready SaaS boilerplate at [shipmvp.tech](https://www.shipmvp.tech) that includes the contract registry and ninth-bug detector as first-class modules with CI integration already wired up, and it has been battle-tested in actual production builds, not just demo repos.
Here is the question that still keeps me up at night: when your test framework does not naturally expose contract signatures, do you annotate them explicitly, infer them from naming conventions, or accept the friction and require a separate registration file? Each approach leaves a different kind of technical debt waiting to collect.
Top comments (0)