Record What Your Code Throws Before You Refactor It

Most refactors keep the happy path intact. They quietly rewrite the error path.

Messages get reworded. Exception types get normalized. Validation order flips. Reviewers read the new structure, not the old failure output.

Freeze those three things before you extract anything.

Why the error path drifts

A helper extraction moves two raise statements into one place. The first error now wins. Nothing fails, tests stay green, and your logs change shape.

Happy-path tests never reach those branches. So they cannot notice.

The fix is not more assertions. It is a recorded baseline of observable failures, replayed after each small change.

Step 1: Put the imperfect module on the table

Here is a realistic messy function. Validation, pricing, and coupon lookup are welded together.

# pricing.py — before
COUPONS = {'WELCOME': 500, 'BULK': 2000}

def quote(qty, unit_cents, coupon=None):
    if qty <= 0:
        raise ValueError('qty must be positive')
    total = qty * unit_cents
    if coupon:
        if coupon not in COUPONS:
            raise KeyError('unknown coupon: ' + coupon)
        total = total - COUPONS[coupon]
    if total < 0:
        raise ValueError('total below zero: %d' % total)
    return total

Note the third branch. It only fires when unit_cents is negative. That is exactly the branch a refactor deletes by accident.

Step 2: Write a malformed-input corpus

Cases are data, not test functions. Keep them in one importable file.

# corpus.py
CASES = [
    {'id': 'qty-zero', 'fn': 'quote', 'args': (0, 100)},
    {'id': 'qty-negative', 'fn': 'quote', 'args': (-2, 100)},
    {'id': 'qty-one', 'fn': 'quote', 'args': (1, 250)},
    {'id': 'coupon-ok', 'fn': 'quote', 'args': (4, 250, 'WELCOME')},
    {'id': 'coupon-unknown', 'fn': 'quote', 'args': (4, 250, 'NOPE')},
    {'id': 'coupon-empty', 'fn': 'quote', 'args': (4, 250, '')},
    {'id': 'negative-unit', 'fn': 'quote', 'args': (1, -100, 'WELCOME')},
    {'id': 'float-qty', 'fn': 'quote', 'args': (1.5, 100)},
]

Four rules made this list short and useful.

1. Two cases per validation branch, one on each side of the boundary.
2. One case per flag state, including empty string and unknown key.
3. One odd-but-legal argument type, such as 1.5.
4. No mocks. Real module, real arguments.

Step 3: Record outcomes and replay them

The harness has two modes. --save writes the baseline. Without it, the run diffs and exits non-zero.

# parity.py
import argparse, importlib, json, re, sys
from pathlib import Path

ADDR = re.compile(r'0x[0-9a-fA-F]+')
PATH = re.compile(r'(?:/[^/ ]+){2,}')

def normalize(text):
    text = ADDR.sub('0xADDR', text)
    text = PATH.sub('<PATH>', text)
    return ' '.join(text.split())

def outcome(fn, args):
    try:
        value = fn(*args)
    except Exception as exc:
        return {
            'kind': 'raise',
            'type': type(exc).__name__,
            'msg': normalize(str(exc)),
            'args': [normalize(repr(a)) for a in exc.args],
        }
    return {'kind': 'return', 'value': normalize(repr(value))}

def record(module_name, cases):
    mod = importlib.import_module(module_name)
    return {c['id']: outcome(getattr(mod, c['fn']), c['args']) for c in cases}

def compare(base, current):
    out = []
    for case_id in sorted(set(base) | set(current)):
        if base.get(case_id) != current.get(case_id):
            out.append({'case': case_id,
                        'before': base.get(case_id),
                        'after': current.get(case_id)})
    return out

def coverage_gaps(module_name, cases, minimum):
    mod = importlib.import_module(module_name)
    public = {n for n, o in vars(mod).items()
              if callable(o) and not n.startswith('_')
              and getattr(o, '__module__', None) == module_name}
    counts = {}
    for c in cases:
        counts[c['fn']] = counts.get(c['fn'], 0) + 1
    return {'uncased': sorted(public - set(counts)),
            'thin': sorted(f for f, n in counts.items() if n < minimum)}

def main(argv=None):
    ap = argparse.ArgumentParser()
    ap.add_argument('--module', required=True)
    ap.add_argument('--corpus', required=True)
    ap.add_argument('--baseline', required=True)
    ap.add_argument('--save', action='store_true')
    ap.add_argument('--min-cases', type=int, default=2)
    a = ap.parse_args(argv)

    cases = importlib.import_module(a.corpus).CASES
    gaps = coverage_gaps(a.module, cases, a.min_cases)
    if gaps['uncased'] or gaps['thin']:
        print('corpus gaps:', json.dumps(gaps))

    if a.save:
        data = record(a.module, cases)
        Path(a.baseline).write_text(json.dumps(data, indent=2, sort_keys=True))
        print('saved', a.baseline, len(data), 'cases')
        return 0

    base = json.loads(Path(a.baseline).read_text())
    breaks = compare(base, record(a.module, cases))
    for item in breaks:
        print(json.dumps(item, sort_keys=True))
    print(f'{len(breaks)} break(s) across {len(base)} cases')
    return 1 if breaks else 0

if __name__ == '__main__':
    sys.exit(main())

Run it once, on the untouched module.

python parity.py --module pricing --corpus corpus --baseline baseline.json --save
python parity.py --module pricing --corpus corpus --baseline baseline.json; echo $?

The saved baseline is boring on purpose. Excerpt:

{
  "coupon-unknown": {"kind": "raise", "type": "KeyError",
    "msg": "'unknown coupon: NOPE'", "args": ["'unknown coupon: NOPE'"]},
  "negative-unit": {"kind": "raise", "type": "ValueError",
    "msg": "total below zero: -600", "args": ["'total below zero: -600'"]},
  "float-qty": {"kind": "return", "value": "150.0"}
}

Commit baseline.json together with parity.py and corpus.py. The baseline is now a contract.

Step 4: Make the smallest safe change

Now extract coupon handling. Resistance is natural, so here is what a first draft looks like.

# pricing.py — first draft, rejected
def _apply_coupon(total, coupon):
    if not coupon:
        return total
    if coupon not in COUPONS:
        raise KeyError(f'unknown coupon {coupon!r}')
    return total - COUPONS[coupon]

def quote(qty, unit_cents, coupon=None):
    if qty <= 0:
        raise ValueError('quantity must be positive')
    total = _apply_coupon(qty * unit_cents, coupon)
    if total < 0:
        raise ValueError(f'total below zero: {total}')
    return total

The structure is what you wanted. The harness exits 1 with three records.

• qty-zero and qty-negative: 'qty must be positive' became 'quantity must be positive'.
• coupon-unknown: "'unknown coupon: NOPE'" became a differently quoted KeyError message.

The fix is to keep the strings byte-identical and merge only the control flow. Rerun. Zero breaks. That is your smallest safe change.

If you want new wording, do it in a second commit that edits the message and updates baseline.json in the same diff. Reviewers then see intent, not drift.

Step 5: Check the corpus for dead weight

A corpus with many cases and few distinct outcomes hides gaps. Group the baseline records by their serialized outcome and inspect the clusters.

This is where a free model is genuinely useful. Paste the baseline JSON and ask which clusters contain cases that cannot fail independently. The model proposes sharper inputs. You decide which cases to add, and you write the expected outcomes yourself.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I use the free model access for that clustering pass and the free server option to host the replay job, so the baseline check runs without occupying my laptop. The corpus and the baseline are plain files, so the workflow survives if you move elsewhere.

Do not let the model write the harness output. It guesses. The recorder measures.

Choosing the right freeze

What you must preserve	Artifact	Why the others miss it
Exception type, message, validation order	exception parity replay	catches branches happy-path tests skip
Exact document or report text	byte-pinned golden file	repr() of a parsed object hides whitespace
Order of writes, emits, or external calls	call-order log with a spy	parity records results, not sequences
Latency or allocation budget	separate measurement run	replay is deterministic only

Limitations

• Normalization hides regressions. Keep the regex list tiny and reviewed in the diff.
• Side-effect order is invisible here. Add a spy log if order matters.
• The corpus is a net with holes. An uncovered branch cannot break parity.
• Dynamic ids and timestamps need explicit normalizers, not broad ones.
• except Exception deliberately misses SystemExit and KeyboardInterrupt.
• This proves behavior did not change. It never proves the behavior is correct.

Who should skip this

Skip it for throwaway scripts, spikes, and code with no observable error contract. Skip it if your team will not commit baseline updates alongside code changes, because the baseline then becomes noise you learn to ignore.

Use it when a module's failures are part of its interface. Record first. Extract second. The two commands above are the whole workflow.