Your Characterization Tests Are Guarding Dead Code: A Coverage Census for Messy Repos

Most characterization tests in a messy repo protect nothing. They pin behavior for functions no entry point ever calls. The suite goes green, the refactor lands, and production breaks in a file nobody characterized.

The fix is not more tests. It is a census before the first test. Classify every public function by how it was actually executed, then characterize only the rows that run.

This post gives you that census: one script, one coverage run per entry point, one decision table. It fits a repo where you must make the smallest safe change and prove it.

1. The two failure modes a census catches

Dead public surface. An exported function has no caller and no test. Write a characterization test and you freeze fictional behavior. Later refactors treat that fiction as a contract.

Test-only surface. A function is reachable only from the suite. Tests pass forever, but the production path takes a different branch. The refactor is safe against the test, not against the app.

Both failures look identical in a coverage percentage. They only separate when you tag coverage by entry point.

2. Write your entry points down as data

Create entrypoints.txt. One line per way a human or scheduler starts the program. Not per module. Not per test.

entry:cli.export       app export --since 2026-10-01
entry:cli.import       app import fixtures/sample.csv
entry:worker.retry     app worker retry --once
entry:api.health       curl -fsS localhost:8000/health

Rules that keep this list honest:

1. Every line must be runnable on a clean checkout.
2. If it needs credentials, stub them now and note the stub.
3. If you cannot run it locally, mark it entry:UNKNOWN instead of guessing.

Four lines of real traffic beat forty imagined use cases.

3. Run each entry point under its own coverage context

coverage.py records a context string per executed line. Set it per run, then append.

# one context per real entry point
COVERAGE_CONTEXT="entry:cli.export"   coverage run --branch -m app export --since 2026-10-01
COVERAGE_CONTEXT="entry:cli.import"   coverage run --branch -a -m app import fixtures/sample.csv
COVERAGE_CONTEXT="entry:worker.retry" coverage run --branch -a -m app worker retry --once

# the suite gets its own namespace, never mixed with entry points
COVERAGE_CONTEXT="test:suite" coverage run --branch -a -m pytest -q tests/

coverage json -o cov.json --show-contexts

Two flags matter. -a appends instead of overwriting. --show-contexts adds a contexts map to the JSON report.

If you use pytest-cov, the equivalent is --cov-context=test. Do not let it overwrite the entry-point data file.

Set relative_files = True in coverage.toml or setup.cfg. Otherwise CI paths and local paths disagree.

4. Intersect coverage with the public surface

A percentage tells you nothing per function. This script does. It parses the source, finds public definitions, and unions the contexts recorded over each body's line span.

#!/usr/bin/env python3
"""census.py: classify public functions by how they were executed."""
import ast
import csv
import json
import sys
from pathlib import Path

SRC = Path(sys.argv[1] if len(sys.argv) > 1 else "src")
COV = json.loads(Path("cov.json").read_text())


def load_contexts(cov):
    """absolute path -> {line number: set of contexts}"""
    by_file = {}
    for path, data in cov["files"].items():
        lines = {}
        for line, ctxs in data.get("contexts", {}).items():
            lines[int(line)] = set(ctxs)
        by_file[str(Path(path).resolve())] = lines
    return by_file


def state(ctxs):
    if any(c.startswith("entry:") and c != "entry:UNKNOWN" for c in ctxs):
        return "LIVE"
    if any(c.startswith("test:") for c in ctxs):
        return "TEST_ONLY"
    return "COLD"


def public_defs(tree):
    for node in ast.walk(tree):
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
            if not node.name.startswith("_"):
                yield node


def main():
    contexts = load_contexts(COV)
    rows = []
    for path in sorted(SRC.rglob("*.py")):
        line_ctx = contexts.get(str(path.resolve()), {})
        tree = ast.parse(path.read_text())
        for node in public_defs(tree):
            end = node.end_lineno or node.lineno
            seen = set()
            for ln in range(node.lineno, end + 1):
                seen |= line_ctx.get(ln, set())
            rows.append({
                "file": str(path),
                "function": node.name,
                "start_line": node.lineno,
                "state": state(seen),
                "contexts": ";".join(sorted(seen)),
            })
    writer = csv.DictWriter(sys.stdout, fieldnames=list(rows[0]))
    writer.writeheader()
    writer.writerows(sorted(rows, key=lambda r: (r["state"], r["file"])))


if __name__ == "__main__":
    main()

Run it and count the rows by state.

python census.py src > census.csv
cut -d, -f4 census.csv | tail -n +2 | sort | uniq -c

A repo at 61% line coverage can easily show 40% of public functions in COLD. That number is the point of the exercise.

5. Read three states, not a percentage

The census output is a work queue. Each state maps to exactly one action before the refactor.

Census state	What it means	Action before the refactor
LIVE	A real entry-point context reached the body	Write one characterization test from a recorded input
TEST_ONLY	Only the test suite reached the body	Read the test. If it asserts a tautology, downgrade to COLD
COLD	No context reached the body in this run	Delete it, or declare it out of scope for this diff

Two refinements keep the table from lying.

UNKNOWN is not COLD. An entry point that needs a production database, a vendor sandbox, or a nightly scheduler produces false COLD rows. List those functions in a skip.txt and mark them UNKNOWN in the PR description. Never characterize an UNKNOWN row.

Decorators run at import time. Line spans cover the body, not the decorator chain. If a decorator holds the real logic, characterize the decorator separately.

6. Characterize only the LIVE rows

Now the tests have a target. One test per LIVE function, minimum. Keep each test small.

1. Capture one real input, from a log line, a fixture file, or a captured HTTP body.
2. Save it under tests/golden/<function>/input.json with the source noted in a comment.
3. Assert on the returned value or the written file, not on internals.
4. Record the exact command that produced it in the test docstring.

# tests/test_export_rows.py
# Input captured from: app export --since 2026-10-01 (2026-10-14 run log)
# Purpose: pin export_rows() before extracting the formatter.
import json
from pathlib import Path
from app.export import export_rows


def test_export_rows_golden():
    fixture = Path("tests/golden/export_rows/input.json")
    expected = json.loads(Path("tests/golden/export_rows/output.json").read_text())
    assert export_rows(json.loads(fixture.read_text())) == expected

Add a second file for fields the refactor is allowed to change. Call it allowed_to_change.json. Anything in that list is a conscious exception, not a surprise diff.

Do not write a characterization test for a COLD row. You would be pinning behavior with no evidence it matters.

7. Gate the diff to the LIVE set

With the census in hand, the smallest safe change becomes mechanical. The refactor may touch LIVE files only.

cut -d, -f1,4 census.csv | grep ',LIVE$' | cut -d, -f1 | sort -u > live_files.txt
git diff --name-only origin/main...HEAD | sort > touched_files.txt
comm -23 touched_files.txt live_files.txt   # must print nothing

If that command prints a path, the diff reached outside the characterized set. Either add a characterization test for that path, or move the change to a follow-up PR.

Two caveats. Paths with commas break cut, so use a small Python one-liner on Windows checkouts. And new files never appear in the census, so treat them as always in scope.

This gate is the whole point. A diff budget says how much you may change. The census says where you may spend it.

8. Where free model access and a free server help

The census itself needs no model. It is deterministic parsing and coverage data.

Two steps around it do benefit. First, entry points that need a reproducible Linux box with the right Python version. Second, drafting candidate inputs for LIVE functions whose recorded traffic is thin. MonkeyCode offers free model access and a free server option, per the operator, and both fit those two steps without a procurement cycle.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What I would not delegate: the state labels and the golden outputs. A generated input is a hypothesis. You still verify it against a real run before it becomes a fixture. A model that invents a plausible input for a COLD function makes the problem worse, not better.

Treat the free server as a place to reproduce your entry points, not as a source of truth. If the box cannot reach the same data, you get UNKNOWN rows, which is a correct answer.

9. Limitations and who should skip this

• Small repos with one entry point. The census collapses to plain coverage. Skip it and characterize everything.
• Repos being rewritten, not refactored. You do not need to freeze behavior you plan to delete.
• Entry points you cannot run at all. Every row becomes UNKNOWN. Fix reproducibility first.
• Coverage contexts on heavy suites. Context data grows fast. Restrict --context to entry points and one suite context.
• C extensions, generated code, and template languages. The AST pass sees nothing. Those stay out of scope explicitly.

The order that survives review

Census first. Tests second. Diff last. Each step produces an artifact a reviewer can check: census.csv, tests/golden/, and an empty comm -23 output.

That is the smallest safe change, stated as evidence instead of confidence. If you want a box to run the entry points on while you build the census, the free server option is a reasonable place to start.