Census Every Raise Site Before You Move a Module

Your test suite checks return values. It rarely checks which exceptions escape. That gap grows when you extract a module.

Here is the rule I use: freeze the error contract first, then move exactly one symbol. Two baselines, one empty diff, one revert command.

The census script below is a template. I have not run it against your repository. Treat its first output as review material, not as truth.

Step 1: Name the boundary and the one symbol

Write the boundary down before touching code. List the public functions that callers use today.

Then pick a single symbol to move. No signature changes. No renames. One commit, one revert target.

boundary: src/reports/parse.py -> exports parse_header, parse_rows
symbol:   parse_header
no-touch: parse_rows, src/reports/emit.py

If you cannot name the symbol in one line, the change is too large.

Step 2: Build a static census of raise sites

Error paths drift because nobody counts them. Count them with the AST, not with grep.

#!/usr/bin/env python3
# tools/error_census.py - template; review output before trusting it
import ast, json, sys
from pathlib import Path

def template_of(node):
    if node is None:
        return ""
    if isinstance(node, ast.Constant) and isinstance(node.value, str):
        return node.value
    if isinstance(node, ast.JoinedStr):
        out = []
        for v in node.values:
            out.append(str(v.value) if isinstance(v, ast.Constant) else "{}")
        return "".join(out)
    return "<dynamic>"

def type_of(exc):
    if exc is None:
        return "<reraise>"
    f = exc.func if isinstance(exc, ast.Call) else exc
    if isinstance(f, ast.Name):
        return f.id
    if isinstance(f, ast.Attribute):
        return f.attr
    return "<dynamic>"

def census(root):
    rows = []
    for path in sorted(Path(root).rglob("*.py")):
        if "/.venv/" in str(path) or "/tests/" in str(path):
            continue
        tree = ast.parse(path.read_text(encoding="utf-8"))
        for node in ast.walk(tree):
            if isinstance(node, ast.Raise):
                msg = node.exc.args[0] if isinstance(node.exc, ast.Call) and node.exc.args else None
                rows.append({"file": str(path), "line": node.lineno,
                             "type": type_of(node.exc), "template": template_of(msg)})
    return sorted(rows, key=lambda r: (r["file"], r["line"]))

if __name__ == "__main__":
    print(json.dumps(census(sys.argv[1]), indent=2))

Run it once and read the output by hand. You will find dead raises and untranslated messages.

python tools/error_census.py src > errors.baseline.json
wc -l errors.baseline.json

Step 3: Snapshot runtime exceptions, not just static ones

Static analysis misses raise exc_cls("boom"). Add a small runtime snapshot for bad inputs.

# tests/test_error_contract.py - characterization, not specification
import json
from pathlib import Path
from reports.parse import parse_header

CASES = {"empty": b"", "garbage": b"\x00\xff", "truncated": b"HDR1|"}

def test_error_contract_is_stable():
    observed = {}
    for name, payload in CASES.items():
        try:
            parse_header(payload)
        except Exception as exc:
            observed[name] = {"type": type(exc).__name__, "msg": str(exc)}
    baseline = json.loads(Path("errors.runtime.json").read_text())
    assert observed == baseline

Generate the baseline once on a clean checkout. Commit both files. They are now the contract.

Step 4: Capture the baseline on a disposable machine

Local baselines lie. Your shell exports, locale, and Python patch version leak into messages.

A throwaway Linux box fixes that. I used MonkeyCode's free server option for this step: fresh checkout, same interpreter, record both baselines there. MonkeyCode also offers free model access, which I used only to draft a first-pass exception taxonomy for human review.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

ssh box 'git clone <repo> && cd repo && python -m venv .venv'
ssh box 'cd repo && .venv/bin/python tools/error_census.py src > errors.baseline.json'
ssh box 'cd repo && .venv/bin/pytest tests/test_error_contract.py --record'
scp box:repo/errors.baseline.json .

Do not upload proprietary code without written approval. Verify current terms yourself; free tiers change.

Step 5: Make the smallest safe change

The move should touch imports and one file. Nothing else.

git switch -c move-parse-header
git mv src/reports/parse.py src/reports/header.py
git commit -m "move: parse_header out of reports.parse"

No logic edits in the same commit. If the diff shows a changed condition, you moved two things.

Step 6: Gate the second commit

Key on basename, type, and template. The move changes paths, so full paths would create false alarms.

# tools/error_gate.py
def key(r):
    return (Path(r["file"]).name, r["type"], r["template"])

def test_no_new_error_sites():
    current = json.loads(run_census("src"))
    baseline = json.loads(Path("errors.baseline.json").read_text())
    allow = {tuple(a["key"]) for a in load_allowlist()}
    added = {key(r) for r in current} - {key(r) for r in baseline} - allow
    assert not added, sorted(added)

Added sites fail the gate. Removed sites pass with a note, because deletions are visible in review.

Step 7: Allow intentional changes with a reason

Some moves legitimately change errors. Make that a written decision, not a silent pass.

[{"key": ["header.py", "ValueError", "bad header magic"],
  "reason": "moved from TypeError; see PR #412",
  "expires": "2026-12-01"}]

An allow-list with an expiry date is a receipt. Without it, the gate decays into comments.

When this gate pays off

| Repo shape | Use the census gate | Why |
| --- | --- |
| Library with public exceptions | Yes | Callers catch types, not messages |
| CLI that users script against | Yes | Exit paths depend on error types |
| Thin CRUD service | Maybe | Few raise sites, low drift risk |
| Heavy dynamic dispatch | No | AST census misses most sites |
| Generated or vendored code | No | Noise drowns the signal |

Pick rows that match your repo. A gate you cannot trust is worse than no gate.

Limitations you should expect

• Dynamic raises resolve to <dynamic>. Read those rows by hand.
• Library-raised exceptions never appear in your source census.
• Runtime snapshots only cover the inputs you listed.
• Basename keys collide in repos with repeated filenames.
• Message text in i18n files sits outside the AST scan.

None of these are fatal. All of them need a sentence in your PR description.

Who should not use this

Skip it if you have no CI to run the gate. Skip it if your team will not maintain an allow-list. Skip it if the module is scheduled for deletion.

Also skip it if you cannot get approval to run code on an outside machine. Run the baseline locally instead, and accept the environment noise.

The habit worth keeping

Count error sites before the move. Snapshot them twice. Require one empty diff.

Then the refactor becomes boring, which is the goal. If you try the census on a single module, the first surprise is usually a raise nobody documented.