Characterization Tests First, Then the Smallest Safe Change

Stop refactoring. Start recording. The smallest safe change wins only if you can prove nothing else moved.

That is the whole method. You freeze the hidden inputs, snapshot the current output, then change one line. The snapshot decides whether you were safe.

This article walks through a runnable loop on a small legacy stand-in module. Swap in your real module and the steps stay the same.

1. Turn the ticket into a behavior, not a design

The ticket here is vague: unknown PLAN breaks invoice generation. Do not translate that into an architecture plan yet.

Translate it into one observable statement. "Unknown plan raises KeyError before any row is processed."

Now the statement is testable. It also tells you the bug lives in one lookup, not in the whole loop.

2. Freeze the hidden globals before you assert anything

Legacy modules read two hidden inputs constantly: the clock and the environment. Both change between your laptop and CI.

Here is the stand-in under test. It is deliberately small and deliberately dirty.

# billing/cycle.py
import datetime
import os

RATES = {"standard": 1.0, "pro": 0.8, "legacy": 1.25}


def invoice_lines(rows):
    today = datetime.date.today()
    plan = os.environ.get("PLAN", "standard")
    rate = RATES[plan]
    out = []
    for row in rows:
        days = (today - row["start"]).days
        if days < 0:
            days = 0
        amount = round(row["units"] * rate * days, 2)
        out.append({"id": row["id"], "days": days, "amount": amount})
    return out

Two hidden inputs sit in four lines. datetime.date.today() reads the machine clock. os.environ.get reads the process environment.

Freeze both by patching the names inside the module under test. Never patch the stdlib module globally.

# tests/test_cycle_char.py
import datetime
import json
import os
import pathlib
import types

import pytest

from billing import cycle

GOLDEN = pathlib.Path(__file__).parent / "golden" / "invoice_lines.json"


class FixedDate(datetime.date):
    @classmethod
    def today(cls):
        return cls(2026, 3, 1)


@pytest.fixture
def frozen(monkeypatch):
    monkeypatch.setenv("PLAN", "pro")
    monkeypatch.setattr(cycle, "datetime", types.SimpleNamespace(date=FixedDate))

This works because the module calls datetime.date.today() by attribute access. A module that does from datetime import date needs a different patch point.

That detail matters. Note it, or you will fight a passing test that proves nothing.

3. Record a golden, then assert against it

Write the rows once, run them, and save the output. The recorder is the source of truth, not your memory of the old behavior.

ROWS = [
    {"id": "a1", "start": datetime.date(2026, 2, 1), "units": 3},
    {"id": "a2", "start": datetime.date(2026, 3, 4), "units": 2},
    {"id": "a3", "start": datetime.date(2026, 3, 1), "units": 0},
]


def test_invoice_lines_matches_golden(frozen):
    got = cycle.invoice_lines(ROWS)
    if os.environ.get("RECORD_GOLDEN") == "1":
        GOLDEN.parent.mkdir(parents=True, exist_ok=True)
        GOLDEN.write_text(json.dumps(got, indent=2))
    assert got == json.loads(GOLDEN.read_text())

Record once with RECORD_GOLDEN=1 python -m pytest tests/test_cycle_char.py -q. The first run always passes, and that is expected.

Read the recorded file by hand before you commit it. A snapshot is not a truth claim.

[
  {"id": "a1", "days": 28, "amount": 67.2},
  {"id": "a2", "days": 0, "amount": 0.0},
  {"id": "a3", "days": 0, "amount": 0.0}
]

Pin the error path too. Empty input still hits the lookup, so the exception fires before the loop.

def test_unknown_plan_raises_keyerror(monkeypatch, frozen):
    monkeypatch.setenv("PLAN", "platinum")
    with pytest.raises(KeyError):
        cycle.invoice_lines([])

4. Prove the harness bites before you trust it

A test that cannot fail is decoration. Break a copy of the code and confirm the suite goes red.

rsync -a --exclude .git ./ /tmp/char-mut/
sed -i 's/if days < 0:/if days < -1:/' /tmp/char-mut/billing/cycle.py
cd /tmp/char-mut && python -m pytest tests/test_cycle_char.py -q; echo "exit=$?"

Expect exit=1. The clamped future date now leaks a negative days value, and the golden catches it.

If the suite still passes, your cases do not cover the branch. Fix that before touching the real module.

5. Take the smallest change, then edit the pin on purpose

Smallest here is one expression. Replace the strict lookup with a guarded one.

-    rate = RATES[plan]
+    rate = RATES.get(plan, RATES["standard"])

This changes observable behavior, so the pinned error test must change with it. Edit it deliberately, in the same commit.

-def test_unknown_plan_raises_keyerror(monkeypatch, frozen):
-    monkeypatch.setenv("PLAN", "platinum")
-    with pytest.raises(KeyError):
-        cycle.invoice_lines([])
+def test_unknown_plan_falls_back_to_standard(monkeypatch, frozen):
+    monkeypatch.setenv("PLAN", "platinum")
+    assert cycle.invoice_lines([]) == []

Now the diff shows one intended behavior edit. Everything else stays green, which is the only proof you have.

Use Python 3.11 or newer here. Python 3.9 is already past its end-of-life date, so new test tooling should not target it.

6. Use a change ladder instead of judgment calls

Rank candidate changes by blast radius. Take rank 0 first, then climb one rung per commit.

Rank	Change	Lines touched	Gate to pass
0	Rename a local variable	1	goldens unchanged
1	Add a guard clause	1-3	goldens unchanged
2	Extract one pure helper	5-15	goldens unchanged, call order identical
3	Change observable behavior	1-5	exactly one pinned test edited on purpose
4	Move the module	many	old import path still pinned
5	Rewrite the module	all	stop, split into ranks 0-4

Rank 3 is the one people skip. They mix a behavior fix with a structure change and lose the ability to review either.

7. Know when to stop the loop

Set stop rules before you start. Seam hunting expands without limits otherwise.

• Stop if freezing the hidden globals takes more than about twenty minutes.
• Stop if the golden output contains wall-clock timestamps or process IDs you cannot stub.
• Stop if behavior depends on network calls, random seeds, or thread interleaving you cannot reproduce.
• Stop if you cannot run the code at all. Characterization without execution is fiction.

Each stop is a signal to shrink scope, not to lower standards.

8. Where model assistance fits, and where it does not

Drafting the edge-case rows in step 3 is the slow part. That is the part worth delegating.

I use MonkeyCode's free model access to propose candidate input rows and hidden-global guesses. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free server option lets me run the mutation check on a repo copy without provisioning a machine of my own.

The model proposes. The recorder decides. Never let a model write the expected output from memory.

Free access and a free server option are the only two availability claims I make. I have not measured throughput or uptime, and a free tier is not an SLA.

9. Limitations and who should not use this

Characterization tests pin current behavior, including bugs. They do not prove correctness and they do not replace intent-based tests.

Exact JSON equality is brittle with floats. Compare amounts with pytest.approx when your golden is hand-edited.

Clock patching only works when the module reads time by attribute access. Import styles, C extensions, and third-party clients may refuse the patch.

Skip this method if you are writing greenfield code, deleting the module next sprint, or have no way to execute it. Skip it if your team cannot review a behavior diff honestly.

If you run this loop on a repo copy, start with the recorder and keep it in charge of the truth.