Your Refactor May Only Touch Lines Your Tests Execute

The rule is one sentence. Every line you add must appear in the coverage report. Not the file. Not the module. The line.

I adopted it after three refactors that passed review and still broke production. Each diff moved code that no test ever entered. Each diff looked small.

Line counts and dependency scores measure the shape of a change. They cannot tell you whether the suite has ever run the code you are touching. Coverage can.

The loop in four steps

Step 1: record coverage on the untouched tree. This is your baseline.

python -m coverage run --branch -m pytest -q
python -m coverage json -o coverage-before.json

--branch is worth the extra time. It records which side of an if actually ran, not just which lines existed.

Step 2: make the smallest change you can defend. Commit it. Do not bundle a rename with a move.

Step 3: record coverage again, on the changed tree.

python -m coverage run --branch -m pytest -q
python -m coverage json -o coverage.json

Step 4: diff against the base branch and let the gate compare both artifacts.

git diff -U0 origin/main...HEAD > change.diff
python tools/covered_edit_gate.py change.diff coverage.json

-U0 matters. With zero context lines, every range in the diff is an added line. No guessing, no hunk arithmetic.

The gate itself

The script is short. It reads two files and answers one question per added line: did the suite execute this line?

# tools/covered_edit_gate.py
import json, re, sys

diff_path, cov_path = sys.argv[1], sys.argv[2]
SKIP_PREFIX = ("tests/", "migrations/", "vendor/")
HUNK = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@")

def added_ranges(diff_text):
    file, out = None, {}
    for line in diff_text.splitlines():
        if line.startswith("+++ b/"):
            file = line[6:].strip()
            out.setdefault(file, set())
        elif line.startswith("@@") and file:
            m = HUNK.match(line)
            start, count = int(m.group(1)), int(m.group(2) or 1)
            out[file].update(range(start, start + count))
    return {f: ls for f, ls in out.items() if ls}

def executed_lines(cov_path):
    files = json.load(open(cov_path))["files"]
    return {p.replace("\\", "/"): set(i["executed_lines"]) for p, i in files.items()}

def is_code(path, lineno):
    try:
        text = open(path, encoding="utf-8").read().splitlines()[lineno - 1]
    except (OSError, IndexError):
        return True
    text = text.strip()
    return bool(text) and not text.startswith("#")

def main():
    diff = open(diff_path, encoding="utf-8").read()
    hit = executed_lines(cov_path)
    failures = []
    for path, lines in added_ranges(diff).items():
        if path.startswith(SKIP_PREFIX):
            continue
        code = {n for n in lines if is_code(path, n)}
        missing = sorted(code - hit.get(path, set()))
        if missing:
            failures.append((path, missing))
    for path, missing in failures:
        print(f"{path}: {len(missing)} added line(s) never executed")
        for n in missing[:10]:
            print(f"  line {n}")
    print("gate: PASS" if not failures else "gate: FAIL")
    sys.exit(1 if failures else 0)

main()

A failing run looks like this. It names the file and the exact lines that no test reached.

$ python tools/covered_edit_gate.py change.diff coverage.json
src/rates.py: 4 added line(s) never executed
  line 88
  line 89
  line 94
  line 97
gate: FAIL

What it refuses, and why that is correct

Change	Gate result	Reason
New line inside a covered function	Pass	The suite runs that code path
New module imported and called by tests	Pass	Post-change coverage lists its executed lines
New module nobody imports	Fail	You added dead code
New if branch never taken	Fail	Unreached branch, use --branch
Blank line, comment, docstring line	Skipped	Not a statement
Deleted lines	Not judged	Deletions need a different check

That last row is honest. The gate constrains additions. Deletions are covered by your suite failing, or by nobody noticing.

A shipping-cost calculator that failed the gate

Our shipping_quote() function had a 70-line body and two call sites. Tests covered the domestic path only. The international branch had zero executed lines for at least a year.

The first extraction attempt moved _zone_rate() out of that branch. The gate failed on four lines. That failure was correct. I had no evidence the branch worked before the move. Any behavior I preserved was imaginary.

So the smallest safe change became a test, not a refactor. I covered the international branch first, confirmed the new test failed when I inverted the branch condition, then ran the extraction again. Second attempt passed. Same three commits as before, one of them just tests.

Where free model access helps, and where it does not

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Covering an uncovered branch is the boring part. I used MonkeyCode's free model access to draft those missing-branch tests from the function signature and a few fixture rows. Roughly half the drafts were usable; the rest asserted the current output without exercising the branch.

Every drafted test went through one manual check. I inverted the branch under test and confirmed the assertion failed. A test that still passes on broken code is decoration.

The nightly run of the full suite is where I used the free server option. I kept the harness single-process with no external services, so it needs nothing but Python and a checkout. Local developers run the same two commands and get the same report.

Limitations you should accept up front

Coverage proves execution, not correctness. A line run once, inside a test with no assertion, passes this gate and proves nothing.

Async callbacks and registered handlers can be defined without ever running. The gate will flag them, and it should.

Generated code and vendored directories must go in SKIP_PREFIX. Otherwise the report fills with noise and people stop reading it.

Large suites make coverage expensive. Run it nightly, then gate pull requests against the cached report.

Who should not use this

Skip this if your repo has no runnable test suite. Build one first; the gate has nothing to read.

Skip it on hotfix branches where speed outranks evidence. Use it on the cleanup branches that keep breaking things.

Skip it in polyglot repos where coverage tooling only half works. A gate that silently passes is worse than no gate.

If you want to test the idea cheaply, run coverage json on your messiest module today. Count how many lines you were about to move are actually executed. That number will change your diff.