Pin the Fault Matrix Before You Refactor a Messy Module

Consider an illustrative composite scenario rather than one specific incident, because the shape repeats across review threads. A service module grows past nine hundred lines, a coding agent returns a four hundred line refactor diff, and the existing suite turns green on the first run. The diff merges, and days later a partner integration fails, because one validation path now raises a bare ValueError where callers expected a typed error with a stable code.

Recent community threads argue loudly about how fast agent-authored diffs arrive, and much less about what those diffs quietly change. The piece that changes first is usually not the happy path but the error surface: exception types, error codes, and the order in which validation rejects a payload. This post pins that surface as data before a messy refactor starts, then constrains the change to the smallest seam available.

Why the Error Surface Decays First

Happy paths get covered because they are easy to observe, cheap to assert, and pleasant to demo in a pull request. Error paths stay thin because each one needs a deliberate fault injection, and their assertions tend to encode prose instead of structure. When a refactor moves validation across layers, the happy path usually survives the move and the exception taxonomy does not.

Three patterns account for most of the drift reviewers see:

• Type flattening — specific errors collapse into ValueError or Exception once a helper is extracted.
• Code loss — the class survives but stops carrying the code attribute that callers branch on.
• Ordering drift — validation runs in a new sequence, so a payload that failed on currency now fails on weight.

All three pass a suite whose only error assertion is "something raised".

Step 1: Enumerate Every Raise Site as Data

Turn the current error surface into a list you can diff, using nothing beyond the standard library. ast.unparse requires Python 3.9 or newer.

# tools/raise_sites.py
"""Print every raise site in a package as file:line:expression."""
import ast
import sys
from pathlib import Path


def raise_sites(path: Path):
    tree = ast.parse(path.read_text(encoding='utf-8'), filename=str(path))
    for node in ast.walk(tree):
        if isinstance(node, ast.Raise):
            if node.exc is None:
                name = 'bare-raise'
            elif isinstance(node.exc, ast.Call):
                name = ast.unparse(node.exc.func)
            else:
                name = ast.unparse(node.exc)
            yield path, node.lineno, name


def main(root: str) -> int:
    rows = []
    for path in sorted(Path(root).rglob('*.py')):
        if '.venv' in path.parts:
            continue
        rows.extend(raise_sites(path))
    for path, line, name in rows:
        print(f'{path}:{line}:{name}')
    print(f'\n{len(rows)} raise sites')
    return 0


if __name__ == '__main__':
    sys.exit(main(sys.argv[1] if len(sys.argv) > 1 else 'src'))

Run it once and commit the output as a baseline artifact, then run it again after the refactor.

python tools/raise_sites.py src > .refactor/raise_before.txt
grep -c . .refactor/raise_before.txt   # example baseline: 47

The raw count is not a quality score, and it should never be optimized for its own sake. What matters is that the diff tells you exactly which raises moved, appeared, or disappeared during the change.

Step 2: Reduce the Raise Sites to a Fault Matrix

Forty-seven raise sites normally collapse into a handful of distinct faults, and the faults are what tests should pin. A shipping rate card gives a compact example.

ID	Injected fault	Expected exception	Stable code	Prose fragment
Z1	zone missing from the rate card	UnknownZoneError	unknown_zone	"unknown zone"
W1	weight absent or None	MissingWeightError	missing_weight	"weight"
C1	parcel currency differs from card	CurrencyMismatchError	currency_mismatch	"currency"
O1	weight above the carrier maximum	OverweightError	overweight	"maximum"

Four rules keep the matrix worth maintaining over several refactors:

1. One row per fault class, not per raise site, since two sites can guard the same contract.
2. Assert on the code column and treat prose as a secondary, replaceable signal.
3. Record which layer raises, because the whole point is detecting a layer move.
4. Keep the matrix in version control beside the tests, so a behavior change is a reviewable diff.

Step 3: Write Characterization Tests That Pin Codes, Not Prose

Keep the tests shaped like the table, so adding one fault stays a one-line change.

# tests/test_error_surface.py
import pytest
from pricing.rate_card import (
    CurrencyMismatchError,
    MissingWeightError,
    OverweightError,
    RateCard,
    UnknownZoneError,
)

# (id, payload, exception class, stable code)
FAULTS = [
    ('Z1', {'zone': 'ZZ', 'weight': 1.0, 'currency': 'USD'}, UnknownZoneError, 'unknown_zone'),
    ('W1', {'zone': 'US', 'weight': None, 'currency': 'USD'}, MissingWeightError, 'missing_weight'),
    ('C1', {'zone': 'US', 'weight': 1.0, 'currency': 'EUR'}, CurrencyMismatchError, 'currency_mismatch'),
    ('O1', {'zone': 'US', 'weight': 99.0, 'currency': 'USD'}, OverweightError, 'overweight'),
]


@pytest.mark.parametrize('case_id,payload,exc,code', FAULTS, ids=[f[0] for f in FAULTS])
def test_error_surface(case_id, payload, exc, code):
    with pytest.raises(exc) as caught:
        RateCard.load().quote(payload)
    assert caught.value.code == code

Parametrized ids keep the output readable, so pytest -q tests/test_error_surface.py reports Z1, W1, C1, and O1 as separate lines. Add one structural guard as well: assert that every exception class in the matrix exposes a non-empty code attribute, which stops a new raise site from shipping without a machine-readable identifier.

Step 4: Make the Smallest Change the Only Allowed Change

With the matrix frozen, give the refactor an explicit churn budget and pick exactly one seam inside it.

git diff --numstat origin/main...HEAD | awk '{a+=$1; d+=$2} END {print a+d " changed lines"}'

Candidate seam	Call sites	In/out types	Side effects	Error paths	Verdict
quote() body	3	dict to Money	none	4	extract one helper
currency formatting	11	str to str	none	1	defer
carrier client wrapper	6	network	I/O	6	do not touch
rate cache	2	dict	TTL clock	2	defer

A seam worth taking has few call sites, no I/O, and error paths already represented in the matrix. Everything else waits for a second pass with its own baseline.

The Verification Gate

Wrap the workflow in one script that CI or a human can run against the branch.

#!/usr/bin/env bash
set -euo pipefail
BASE=${1:-origin/main}
BUDGET=${2:-40}

python tools/raise_sites.py src > /tmp/raise_after.txt
diff -u .refactor/raise_before.txt /tmp/raise_after.txt || true

pytest -q tests/test_error_surface.py tests/test_value_characterization.py

CHURN=$(git diff --numstat "$BASE"...HEAD | awk '{a+=$1; d+=$2} END {print a+d}')
echo churn=$CHURN budget=$BUDGET
test "$CHURN" -le "$BUDGET"

The raise-site diff stays informational, while the two hard gates are the error-surface tests and the churn limit. Treat any new raise site as a contract change: either add a matrix row in the same commit, or move the raise instead of introducing it.

Where an Agent Fits, and Where It Does Not

Drafting matrix rows across dozens of raise sites is mechanical work, which is the kind of task a coding agent can genuinely accelerate. This workflow uses MonkeyCode for that step, where the free model access covers the drafting and the free server option gives the gate somewhere to run without local provisioning. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Scope the prompt tightly, and ask the agent to read .refactor/raise_before.txt, cluster the raise sites by exception name and payload condition, propose one matrix row per cluster with a candidate code, and then stop. Verify every proposed code against one real runtime call before freezing it, because the agent cannot know your taxonomy intent.

Three things the agent should never be trusted with:

• Inventing codes that the exception classes do not actually expose.
• Merging two faults into one row because their payloads look superficially similar.
• Editing the expectations after a failing run, which inverts the purpose of the matrix.

If you want to try the gate end to end without provisioning a machine, the free model access and free server option are enough to run the script, the tests, and the churn check.

Limitations and Who Should Skip This

• Code assertions require errors that actually carry codes; prose matching is wording- and locale-dependent and breaks for the wrong reasons.
• Coverage here means "raise sites referenced by at least one matrix row", which is a proxy for completeness rather than proof of safety.
• Concurrency, timing, and performance behavior sit outside this workflow, and refactors that move work across threads need separate evidence.
• The churn budget is a smell detector, not a correctness check, so a small diff can still break a contract and a large legitimate move needs a deliberate budget raise.
• Skip this entirely if your errors are user-facing prose only, or if the module has no taxonomy worth preserving, since the ceremony would protect nothing.

Messy refactors rarely fail because the happy path changed, and they fail quietly because the error contract moved. Freeze the fault matrix first, take one seam with a churn budget, and let the gate make "the tests still pass" mean something specific about behavior.