Do Not Split a Validator Until Exception Types Are Gold

Do not split a validator until exception types are gold. Characterization tests lock class, args, and attributes first. The smallest safe change comes after that snapshot.

The bug this order prevents

Messy validators often raise three different exception types today. A helper extract can swap ValueError for TypeError overnight. Callers that catch one class will miss the other.

Return values look stable while raise paths silently rotate. Tests that only assert happy paths will stay green. Production catch blocks then fail on the next deploy.

Python treats bool as a subclass of int. A range extract can accept True by accident. The original TypeError for True must stay pinned.

What counts as the raise path

Pin the exception class name before any code move. Pin the args tuple and selected public attributes next. Do not pin tracebacks because they are not contracts.

Store the message string as a weak diagnostic field. Fail the suite on class or args drift only. Message edits are common during later cleanup work.

Record cause class names when chaining is part of behavior. Skip cause data when the original code never chains. The snapshot must match the current messy module.

Artifact: raise-path gold harness

This harness dumps one failing input as JSON gold. Keep the gold file next to the pytest module. Treat the file as read-only until you accept drift.

Label the module below as a fixture example. It is not a live production dump. Copy the shape, then paste your real function.

# fixture example: messy_validate.py

def validate_qty(value):
    if isinstance(value, bool):
        raise TypeError("bool is not qty")
    if not isinstance(value, int):
        raise TypeError("qty must be int")
    if value < 1:
        raise ValueError("qty below 1")
    if value > 99:
        raise ValueError("qty above 99")
    return value

# test_raise_path_gold.py
from __future__ import annotations

import json
import os
from pathlib import Path
from typing import Any, Callable

from messy_validate import validate_qty

GOLD = Path(__file__).with_name("raise_path_gold.json")
STRICT = ("raised", "class", "module", "args", "errno", "cause_class")

CASES = [
    {"id": "bool_true", "args": [True]},
    {"id": "bool_false", "args": [False]},
    {"id": "str_qty", "args": ["12"]},
    {"id": "none_qty", "args": [None]},
    {"id": "zero", "args": [0]},
    {"id": "negative", "args": [-3]},
    {"id": "too_big", "args": [100]},
]


def dump_raise(exc: BaseException) -> dict[str, Any]:
    cause = exc.__cause__
    return {
        "raised": True,
        "class": type(exc).__qualname__,
        "module": type(exc).__module__,
        "args": [repr(a) for a in exc.args],
        "errno": getattr(exc, "errno", None),
        "cause_class": None if cause is None else type(cause).__qualname__,
        "message": str(exc),
    }


def record_case(fn: Callable[..., Any], *args: Any) -> dict[str, Any]:
    try:
        fn(*args)
    except BaseException as exc:
        return dump_raise(exc)
    return {
        "raised": False,
        "class": None,
        "module": None,
        "args": [],
        "errno": None,
        "cause_class": None,
        "message": None,
    }


def build_gold() -> dict[str, Any]:
    return {case["id"]: record_case(validate_qty, *case["args"]) for case in CASES}


def load_gold() -> dict[str, Any]:
    return json.loads(GOLD.read_text(encoding="utf-8"))


def strict_view(payload: dict[str, Any]) -> dict[str, Any]:
    return {key: payload.get(key) for key in STRICT}


def test_raise_path_matches_gold() -> None:
    current = build_gold()
    if os.environ.get("UPDATE_GOLD") == "1":
        GOLD.write_text(json.dumps(current, indent=2, sort_keys=True) + "\n", encoding="utf-8")
    assert GOLD.exists(), "gold missing; run UPDATE_GOLD=1 once"
    expected = load_gold()
    assert set(current) == set(expected), "case ids drifted"
    for case_id in sorted(current):
        assert strict_view(current[case_id]) == strict_view(expected[case_id]), case_id

Each case must name the input and the function under test. Do not generate inputs from a model first. Copy real failing payloads from logs or tickets.

grep -nE 'raise |Error\(|Exception\(' messy_validate.py
UPDATE_GOLD=1 pytest test_raise_path_gold.py -q
git add messy_validate.py test_raise_path_gold.py raise_path_gold.json
pytest test_raise_path_gold.py -q

Commit the gold file before any validator extract begins. A later gold diff is the only allowed contract change. If gold changes, the extract is not smallest.

Workflow

1. Inventory raise sites

Search the messy module for raise and exception constructors. Record the line numbers in a short checklist file. Leave unchecked sites out of the first extract.

2. Build the case table

Map each failing input to one expected raise path. Prefer production samples over synthetic edge-case guesses. One input per gold record keeps diffs readable.

Include the bool inputs even when they look redundant. True is an int subclass in CPython. Missing that case lets a later extract accept True.

3. Freeze gold

Run pytest once and write raise_path_gold.json. Commit tests, gold, and the messy module together. That commit is the baseline for later extracts.

4. Stub asserts from gold, not from a rewrite

Feed the gold file to a coding model after freeze. MonkeyCode offers free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free server can run the pytest gold harness. The free model can propose assert stubs from JSON only. Reject any patch that edits the validator there.

5. Extract one check

Move a single predicate into a new helper function. Keep the original exception class and args unchanged. Re-run the gold suite before any second extract.

# smallest change example — range only, types stay put

def _qty_in_range(value: int) -> None:
    if value < 1:
        raise ValueError("qty below 1")
    if value > 99:
        raise ValueError("qty above 99")


def validate_qty(value):
    if isinstance(value, bool):
        raise TypeError("bool is not qty")
    if not isinstance(value, int):
        raise TypeError("qty must be int")
    _qty_in_range(value)
    return value

Keep the bool check ahead of the range helper. Reordering those guards is a second change. Gold must stay byte-identical on strict keys.

6. Read the gold diff

A class rename is a failed extract, not cleanup. An args tuple change is also a failed extract. A message-only diff can wait for a later commit.

pytest test_raise_path_gold.py -q
git diff -- raise_path_gold.json

Empty gold diff means the extract kept the raise path. Any strict-key diff means revert the helper immediately. Recut the extract until the gold file is silent.

Decision table

Use this table when a gold test fails after the extract. Do not negotiate the class column as optional. Message warnings stay local to the test helper.

Observed drift	Fail gold?	Allowed in the smallest change?
Exception class or module	Yes	No
args tuple via repr	Yes	No
errno or other pinned attr	Yes	No
__cause__ class name	Yes, if originally chained	No
Message text only	No, warn in helper	Yes, later commit
Traceback text	No	Yes
Happy-path return value	Out of scope here	Separate snapshot

What this workflow does not prove

Gold raise paths do not prove the validator is correct. They only prove the error contract did not drift. Behavioral bugs in the messy module stay in place.

Locale and translation layers can rewrite message strings. Do not fail CI on translated message text. Keep class and args as the stable comparison keys.

Exception subclasses can share a catch-all parent class. Pin the exact qualname, not a parent isinstance check. An extract that widens the type is still drift.

Multiple threads can interleave validation on shared state. This harness is single-threaded and input-driven by design. Do not use it as a concurrency proof.

Who should not use this approach

Skip this workflow for crypto and auth failure messages. Those paths may require intentional message instability on purpose. Security reviews still need threat analysis beyond gold JSON.

Skip it when the module has no failing inputs yet. Characterization without cases only freezes empty behavior today. Write the case table first, then dump gold.

Skip it if the team cannot run pytest locally. The gold file is useless without a repeatable runner. A one-off notebook dump will not catch later drift.

Limitations

The harness stores args with repr, not deep equality. Objects without stable repr will produce noisy gold diffs. Convert those args to a hand-chosen canonical form.

Custom exceptions with unlisted attributes will evade the dump. Extend dump_raise when a public field is load-bearing. Private underscore fields stay out of the contract.

The workflow slows extracts that should be mechanical moves. That cost is the point of the smallest change rule. Speed returns after gold exists for the messy module.

Keep raise-path gold in git beside the validator. Extract one predicate only after the suite is green. Class drift means revert, then recut the helper.

A free server is enough to rerun this gold suite.