Characterization Tests First: Record the Behavior, Then Make One Safe Change

Characterization tests record what the code does today, bugs included.
They are a baseline, not a specification.
So the order matters: record, freeze the noise, then change one thing.

This is the workflow I use on repos where nobody remembers the original intent.
It works on a 4,000-line module as well as a 200-line one.
The only requirement is a reproducible entry point.

Why messy-repo refactors fail without a baseline

Most refactors start with a reading of the code, not a recording of it.
Reading tells you what you think the code does.
A transcript tells you what it actually printed, wrote, and returned.

Hidden callers make this worse.
A "private" helper is often imported by a script nobody owns.
Without a baseline, you cannot tell a bug fix from a regression.

Step 1 — Record one entry point before touching anything

Pick the narrowest seam you can execute.
Prefer a CLI or an importable function over an internal method.
Record real behavior: return code, stdout, stderr tail, and files written.

# tests/test_char_quote.py
import json, os, re, subprocess, sys, pathlib
import pytest

ROOT = pathlib.Path(__file__).resolve().parents[1]
FIX = pathlib.Path(__file__).parent / "fixtures"
APPROVED = pathlib.Path(__file__).parent / "approved"

TS = re.compile(r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(\.\d+)?")
TMP = re.compile(r"/tmp/[A-Za-z0-9_\-]+")


def normalize(raw: str) -> str:
    raw = TS.sub("<TS>", raw)
    raw = TMP.sub("<TMP>", raw)
    return raw


def capture(case: pathlib.Path, workdir: pathlib.Path) -> dict:
    env = dict(
        os.environ,
        PYTHONPATH=str(ROOT),
        TZ="UTC",
        LC_ALL="C",
        PYTHONHASHSEED="0",
        QUOTE_CLOCK="2020-01-01T00:00:00Z",
        QUOTE_SEED="1234",
    )
    proc = subprocess.run(
        [sys.executable, "-m", "legacy.quote", "--case", str(case)],
        capture_output=True, text=True, env=env, cwd=workdir, timeout=30,
    )
    return {
        "returncode": proc.returncode,
        "stdout": normalize(proc.stdout).splitlines(),
        "stderr_tail": normalize(proc.stderr).splitlines()[-5:],
        "files": sorted(
            normalize(str(p.relative_to(workdir)))
            for p in workdir.rglob("*") if p.is_file()
        ),
    }

The driver runs in a temp cwd, so written files are observable.
PYTHONPATH keeps imports working while cwd stays disposable.
One run gives you one JSON object you can diff forever.

@pytest.mark.parametrize("case", sorted(p.name for p in FIX.glob("*.json")))
def test_characterization(case, tmp_path):
    got = capture(FIX / case, tmp_path)
    target = APPROVED / f"{case}.json"
    if os.environ.get("CHAR_UPDATE") == "1":
        target.write_text(json.dumps(got, indent=2, sort_keys=True))
        pytest.skip(f"recorded {target.name}")
    assert got == json.loads(target.read_text()), target.name

Step 2 — Normalize the noise, not the behavior

A raw transcript is full of values you do not care about.
Normalize only what is provably non-behavioral.
Everything else stays in the snapshot, mismatched and loud.

Rules that have held up for me:

1. Replace timestamps and temp paths with stable tokens.
2. Sort collections that have no defined order.
3. Round floats only at the serialization boundary, never in the code under test.
4. Keep stderr's last lines; full stderr is usually too noisy to review.

Over-normalizing is the dangerous direction.
If you strip numeric output, the snapshot passes while the math is wrong.

Step 3 — Freeze every input you cannot control

Unfrozen inputs turn a baseline into a coin flip.
Set these before the first recording:

TZ=UTC LC_ALL=C PYTHONHASHSEED=0 \
  pytest -q tests/test_char_quote.py

PYTHONHASHSEED matters for anything iterating a set.
Timezone and locale matter for formatting and date math.
For time itself, inject a clock or freeze it; a patched now() is checked in code.

Randomness needs a seed that the test passes in.
If the module reads a global RNG, add the seed parameter first.
That change is unverified, and you should commit it alone.

Step 4 — Run the baseline twenty times before trusting it

A flaky baseline is worse than no baseline.
It teaches reviewers to ignore diffs.

for seed in $(seq 1 20); do
  PYTHONHASHSEED=$seed pytest -q tests/test_char_quote.py || break
done

Any failure here is a finding, not an inconvenience.
Fix the nondeterminism before you refactor anything.
Otherwise every later failure has two possible explanations.

Step 5 — Make the smallest safe change

One change per commit, transcript unchanged each time:

1. Rename or reorder. Re-run the transcript. Commit.
2. Extract one pure function from the messy method. Re-run. Commit.
3. Inject the clock, path, or RNG as a parameter. Re-run. Commit.
4. Delete the normalization rule the injection made unnecessary. Re-run. Commit.
5. Only now change behavior, with a new test that states the intent.

Step 4 is the one people skip.
Shrinking the normalizer is how you prove the seam actually improved.

Decision table: how much verification each change needs

Change	Transcript expectation	Evidence strength
Rename local, reorder imports	Identical	Strong
Extract a pure function	Identical	Strong
Inject clock or path	Identical, fewer normalizers	Strong
Change rounding or output format	Changes on purpose	Needs human review
Replace the algorithm	Identical, but weak proof	Weak; add real tests

Where a coding assistant actually helps

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two parts of this loop are tedious but low-judgment.
The first is inventing input cases from signatures and docstrings.
The second is running the 20× loop while you write the normalizers.

I used MonkeyCode's free model access to draft candidate fixtures.
I then edited them by hand, because the transcript is the authority, not the model.
I ran the repeated loops on the free server option, since those runs are long and parallel.
Whether free model access or a free server suits your setup depends on your own constraints.
Treat both as operator-stated availability, not as a guarantee of capacity or permanence.

A model is useful for volume here, and useless for truth.
It has no idea which printed value is a bug.

Limitations, and who should not do this

Characterization tests freeze bugs as faithfully as they freeze features.
They tell you what is, never what should be.

Skip this approach if the module is scheduled for deletion.
Skip it if the output is random by contract and callers already tolerate that.
Skip it if there is no reproducible entry point you can drive from a test.

If you must build a driver first, know that the driver is an unverified change.
Keep it as thin as possible and review it twice.
Also watch snapshot size: huge golden files get rubber-stamped in review.

Checklist before you commit the baseline

• One entry point, executed for real, not mocked.
• Timestamps, temp paths, and ordering normalized.
• Timezone, locale, hash seed, and RNG seed pinned.
• Twenty runs, all identical, before any refactor commit.
• Every subsequent commit keeps the transcript byte-identical.

If you try this on one messy module, run the twenty times first.