Seed Faults Before the Smallest Safe Change: A Gate for Refactoring a Messy Module

You inherit a module that has outlived three owners, and its tests are green on your laptop but red in CI for reasons nobody has written down. The instinct is to write characterization tests, watch them pass, and start refactoring the same afternoon. That sequence feels disciplined, and it is exactly how teams ship a cleanup that quietly changes behavior in a branch no test ever executes.

A passing suite is not evidence that the suite can detect change. It is evidence that the suite agrees with the code as it exists right now, which is a much weaker claim. The gate below fixes that gap by attacking your own tests before you touch the module, using a small set of deliberately broken patches that the suite must reject.

Why a green characterization suite proves so little

Characterization tests are written from observed output, so they inherit every blind spot of the person who observed it. If you only ever ran the module with valid payloads, your tests will encode valid payloads and nothing about the error path. If you captured results but not call order, a refactor can move a side effect across a validation boundary and every assertion still passes.

There is a second, quieter problem: the machine that froze the behavior is usually the machine that validates the refactor. Local caches, stale build artifacts, and a globally installed dependency can all make a broken change look correct. Two environments that disagree are more informative than one environment that agrees with itself.

The gate below addresses both failure modes with three phases, and each phase produces an artifact you can review in a pull request.

The three-phase gate

1. Freeze the current behavior into committed snapshots, including error paths and ordering where it is observable.
2. Falsify the frozen suite by applying seeded faults in an isolated copy and requiring every fault to turn the suite red.
3. Thaw one seam at a time, and re-run the same falsification step after each accepted change.

Phase 2 is the part most teams skip, and it is the only phase that tells you whether Phases 1 and 3 mean anything.

Phase 1: freeze behavior, not intentions

Pin outputs as files rather than inline literals, so a diff in behavior shows up as a readable diff in review. The command below uses syrupy-style snapshot flags; substitute your own snapshot plugin if you use a different one.

# Freeze the behavior inventory for one module and commit it as an artifact
python -m pytest tests/characterization/pricing -q --snapshot-update
git add tests/characterization/pricing && git commit -m "freeze: pricing behavior inventory"

Two rules keep this phase honest. First, capture inputs that reach the awkward branches, including None, empty collections, and the boundary values of every tier or threshold. Second, capture what the module touches, not only what it returns, when the caller can observe the difference.

Phase 2: draft the suite with a second reader, then attack it

A useful trick is to let a model that did not write the module draft additional characterization cases from the diff history and the public surface. It has no memory of your assumptions, so it tends to propose the boring edge cases you skipped. I used MonkeyCode's free model access for this drafting pass, and its free server option for the clean-machine rerun described below, because both were available without procurement.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat drafted tests as candidates, never as truth. Every generated case must pass against the unmodified code before it joins the frozen suite, and any case that fails is either a real bug or a misunderstanding of the contract. The falsification step below is what separates the two.

Phase 3: seed faults and require red

The script is a template. It copies the repository into a temporary directory, applies one patch per seeded fault, runs the frozen suite, and reports any patch that the suite fails to catch.

#!/usr/bin/env python3
"""seed_faults.py - prove the frozen suite can actually fail.
Template: adapt SUITE_CMD, PATCH_DIR and the ignore list to your repo.
"""
import shutil, subprocess, sys, tempfile
from pathlib import Path

REPO = Path(".").resolve()
PATCH_DIR = REPO / "faults"                      # one *.patch per seeded fault
SUITE_CMD = [sys.executable, "-m", "pytest", "-q", "-x"]
IGNORE = shutil.ignore_patterns(".git", "node_modules", "__pycache__", ".venv")

def run_suite(root: Path) -> int:
    return subprocess.run(SUITE_CMD, cwd=root, capture_output=True).returncode

def main() -> int:
    patches = sorted(PATCH_DIR.glob("*.patch"))
    if not patches:
        print("no faults found; this gate is not armed")
        return 2
    survivors = []
    for patch in patches:
        with tempfile.TemporaryDirectory() as tmp:
            work = Path(tmp) / "repo"
            shutil.copytree(REPO, work, ignore=IGNORE)
            applied = subprocess.run(
                ["git", "apply", str(patch)], cwd=work,
                capture_output=True, text=True,
            )
            if applied.returncode != 0:
                print(f"[skip]     {patch.name}: patch did not apply")
                print(applied.stderr.strip())
                survivors.append(patch.name)
                continue
            killed = run_suite(work) != 0
            print(f"[{'killed' if killed else 'SURVIVED'}] {patch.name}")
            if not killed:
                survivors.append(patch.name)
    total = len(patches)
    print(f"\n{total - len(survivors)}/{total} seeded faults killed")
    return 1 if survivors else 0

if __name__ == "__main__":
    raise SystemExit(main())

Run it before the refactor and after every accepted step:

python seed_faults.py            # exit 1 means the suite has holes
python -m pytest tests/characterization/pricing -q

Exit code 1 is the signal you want from a suite you have not yet touched, because a survivor is a behavior your tests cannot defend.

Which faults are worth seeding

You do not need hundreds of patches. A dozen, each representing a mistake a real refactor makes, will expose most of the holes in a first-pass suite.

Fault class	Example patch	What a kill proves
Boundary shift	> becomes >= in a discount tier	the suite exercises the edge, not just the middle
Constant drift	change a rounding mode or currency precision	golden values are pinned, not recomputed from the implementation
Order change	move a write before validation	the suite observes sequence, not only final results
Error contract	raise a different exception type	failure paths have witnesses
Default fallback	delete a config default	the fallback branch is covered
Silent widening	accept a string where an enum was expected	input validation is asserted, not assumed

Keep each patch minimal and single-purpose, and store it next to the tests so the next owner can see why it exists. When a refactor legitimately changes behavior, delete or rewrite the matching fault in the same commit and explain the change in review.

Then make the smallest safe change

With the suite armed, the refactor becomes a ranking problem rather than a creative one. Score each candidate change by how many frozen tests cover the touched lines, how cheaply it can be reverted, and whether it alters an externally observed contract.

Candidate change	Coverage of touched lines	Revert cost	Verdict
Rename a private helper	full	one commit	proceed
Extract a pure calculation into its own function	full	one commit	proceed
Reorder two side effects	partial	needs data restore	proceed only behind a flag
Replace a hand-rolled date path with a library	partial	one commit	defer until a test exists
Change a rounding rule	none	breaks stored values	do not couple to this refactor

Two practices keep the ladder short. Ship one rung per commit, and re-run the fault seeding as part of the same review so the reviewer sees the evidence rather than the claim. If a rung requires more than three files of unrelated cleanup, it is two rungs.

Run the loop on a machine that did not freeze the behavior

The clean-machine rerun is where the free server option earns its place in this workflow. Copy the frozen suite and the fault patches to a fresh instance, install from the lockfile, and run both commands there. This catches the dependency that only exists in your global environment and the artifact that only builds on your OS image.

Use the result as a correctness signal, not as a performance number. Shared or bursty hardware tells you nothing trustworthy about latency, so do not quote timings from that run as benchmarks.

Limitations and who should skip this

• Free tiers are terms, not guarantees. Availability, rate limits, and retention rules can change; read them before you build a permanent pipeline on them.
• Drafted tests inherit model blind spots. Fault seeding only catches faults you thought to write, so a survivor list is a lower bound on risk, not a proof of safety.
• Hosted drafting has a confidentiality cost. Check your organization's rules before sending proprietary source to any hosted model, free or paid.
• The script is a template. Paths, suite commands, and the ignore list need adapting, and monorepos usually need per-package runs.

Skip this approach if you maintain a single-owner script with no callers, if the repository cannot be installed reproducibly from a lockfile, or if your generated tests require formal sign-off that no one on the team can give. In those settings, a plain smoke test and a manual diff review are more honest than a ceremony that cannot fail.

If you want to try the falsification phase before committing any infrastructure, MonkeyCode's free model access can draft a first batch of candidate cases and its free server option can host the clean rerun, which is enough to run the loop once end to end on a module you already distrust.