Prove Reach Before You Move: A Coverage Gate for Characterization Tests

A characterization test only protects the lines it executes.

If your test never reaches the branch you plan to move, it cannot tell you the move broke it. That sounds obvious. It is also the most common reason messy-repo refactors roll back.

This post describes one gate. Measure reach first. Then make the smallest safe change.

The rule

Before editing, declare the exact line range you intend to change.

Then prove that a test executes every line in that range.

If the gate fails, you have two choices. Add a case, or shrink the range.

There is no third option called refactor anyway.

Step 1: Turn the seam into line numbers

Intuition drifts between reviewers. Line numbers do not.

Get the range with a twelve-line helper.

# seam.py - print the line range of one function
import ast
import sys

path, name = sys.argv[1], sys.argv[2]
tree = ast.parse(open(path).read())

for node in ast.walk(tree):
    if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and node.name == name:
        print('%d-%d' % (node.lineno, node.end_lineno))
        break
else:
    sys.exit('seam.py: %s not found in %s' % (name, path))

Run it against the function you want to move.

$ python seam.py billing/render.py render_invoice
214-268

Record that range in the PR description. It becomes the definition of small change for this refactor.

Step 2: Draft the characterization tests

The goal here is not elegance. The goal is observability.

Feed the tests real payloads from logs, fixtures, or a captured run. Assert on things you can compare later: strings, dicts, status codes, exception types.

Drafting ten of these tests by hand is slow and boring. This is the one place where free model access genuinely helps. Draft the boilerplate shape on MonkeyCode, then delete what you cannot defend in review.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat drafted tests as inputs, not evidence. A test you did not read is a liability with a green checkmark.

Step 3: Measure reach with coverage

Coverage is the cheapest available proxy for reach.

coverage run -m pytest tests/test_render_char.py -q
coverage json -o coverage.json

Now apply the gate. The script reads the coverage JSON, expands your planned range, and exits non-zero on any line the tests never touched.

# reach_gate.py - fail if planned lines are never executed
import argparse
import json
import sys


def parse_ranges(spec):
    out = []
    for part in spec.split(','):
        part = part.strip()
        if not part:
            continue
        if '-' in part:
            lo, hi = part.split('-', 1)
            out.append((int(lo), int(hi)))
        else:
            out.append((int(part), int(part)))
    return out


def expand(spec):
    lines = set()
    for lo, hi in parse_ranges(spec):
        lines.update(range(lo, hi + 1))
    return lines


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument('--coverage', default='coverage.json')
    ap.add_argument('--file', required=True)
    ap.add_argument('--lines', required=True)
    ap.add_argument('--allow-missing', default='')
    args = ap.parse_args()

    with open(args.coverage) as fh:
        report = json.load(fh)

    entry = report['files'].get(args.file)
    if entry is None:
        print('reach-gate: no coverage entry for %s' % args.file)
        return 2

    executed = set(entry['executed_lines'])
    planned = expand(args.lines)
    allowed = expand(args.allow_missing)
    uncovered = sorted(planned - executed - allowed)

    print('reach-gate: planned=%d reached=%d allowed=%d uncovered=%d' % (
        len(planned), len(planned & executed), len(planned & allowed), len(uncovered)))

    if uncovered:
        print('uncovered: ' + ','.join(str(n) for n in uncovered))
        return 1
    return 0


if __name__ == '__main__':
    sys.exit(main())

A failing run looks like this. The exit code matters more than the message.

$ python reach_gate.py --file billing/render.py --lines 214-268
reach-gate: planned=55 reached=41 allowed=0 uncovered=14
uncovered: 226,227,231,236,240,244,247,251,255,260,262,264,266,268
$ echo $?
1

Step 4: Decide what to do with the gaps

Fourteen uncovered lines in a fifty-five line range is normal. Do not panic. Classify them.

Situation	Action
Defensive branch you cannot trigger	Allow-list it with a comment and a follow-up issue
Needs infra you do not have (SMTP, S3, clock)	Fake the boundary, keep the branch
Genuinely unreachable	Delete it in a separate commit before the refactor
Belongs to another function	Shrink the range, you drifted

Prefer shrinking over allow-listing. A smaller covered range beats a larger excused one.

Step 5: Freeze outputs before you edit

Coverage proves reach. It does not prove sameness. You need a second artifact.

This harness runs every case and writes a normalized snapshot.

# tests/char_harness.py
import json
import sys

from billing.render import render_invoice
from tests.char_cases import CASES


def normalize(value):
    if isinstance(value, dict):
        return {k: normalize(value[k]) for k in sorted(value)}
    if isinstance(value, (list, tuple)):
        return [normalize(v) for v in value]
    if isinstance(value, set):
        return sorted(normalize(v) for v in value)
    return value


def snapshot():
    out = {}
    for name, payload in CASES:
        try:
            out[name] = {'ok': True, 'value': normalize(render_invoice(payload))}
        except Exception as exc:
            out[name] = {'ok': False, 'error': type(exc).__name__}
    return out


if __name__ == '__main__':
    with open(sys.argv[1], 'w') as fh:
        json.dump(snapshot(), fh, indent=2, sort_keys=True)

Capture the baseline before touching the file.

python -m tests.char_harness goldens/render_before.json
git add goldens/render_before.json

Note the deliberate omission: exception messages are dropped, types are kept. Messages drift across versions. Types are the contract.

Step 6: Make the smallest change, then rerun both gates

Smallest means the diff touches only lines inside the gated range. Anything outside that range stops the work.

Line numbers shift after the edit, so recompute the range first.

python seam.py billing/render.py render_invoice
python reach_gate.py --file billing/render.py --lines 205-259
python -m tests.char_harness goldens/render_after.json
diff -u goldens/render_before.json goldens/render_after.json

An empty diff means behavior held on the cases you chose. A reach failure after the move means you changed control flow more than you planned. Treat that as a signal, not a formality.

If you would rather not install this toolchain locally, the free model access and free server option on MonkeyCode will run the same loop: draft tests, execute the gate, keep the artifacts in the repo. The gate logic is yours either way.

Where this gate is weak

Coverage measures reach, not assertion strength. A test can execute a line and assert nothing about it. Pair this with a review pass or an injected-fault run.

Line ranges are brittle. Any upstream commit invalidates them. Recompute on every branch, not once per quarter.

Golden files rot. They encode today's bugs on purpose. Label them as characterization, never as specification.

Time, randomness, and concurrency destabilize outputs. Normalize those fields explicitly or exclude the cases and say so.

The tooling here is Python-specific. The idea ports to other languages; the commands do not.

Who should skip this

Code with no stable observable output. If nothing comes out of the function, there is nothing to freeze. Add boundary logging first.

Refactors that are really rewrites. A new module with new behavior is a parity problem you cannot solve, and you should not pretend otherwise.

Suites too slow to run twice per change. A gate that takes twenty minutes gets bypassed by Thursday.

The short version

Write down the lines. Prove the tests reach them. Freeze the outputs. Change one thing. Rerun both gates.

That order is the entire method. Everything else is commentary.