Characterize the Report Boundary Before One Safe Change

Messy modules fail when the first edit is a rewrite. Pin the public report using characterization tests first. Then change one function, not the whole tree.

This workflow targets tangled scripts, not greenfield services. You freeze outputs at the public boundary only. You never guess intended behavior from helper names.

The failure you are preventing

AI-assisted edits often rewrite three files together. New tests then assert the rewritten shape. The old contract vanishes without a recorded baseline.

That pattern is not a true behavior-preserving refactor. It is still an unreviewed product change. Characterization tests exist to block that mode.

A messy report script mixes CSV, cache, and files. A coding model proposes a clean class design. You accept the patch because fresh tests pass.

Those tests never saw the previous output bytes. Duplicate SKUs may now collapse to one. Share percents may now use integer division.

Users still submit last week's CSV files. Their summary.txt no longer matches the saved archive. You cannot prove when the silent drift started.

What you pin at the boundary

Pin three artifacts from a single public call. Pin stdout text as exact output characters. Pin each written file as exact bytes.

Also pin the return payload and cache map. Do not pin private helper function names here. Do not pin call order inside one function.

The public boundary remains the only frozen contract. Internal structure may still move in later passes. Golden files must survive that internal movement intact.

The messy fixture

The sample below is a labeled fixture. It is not a recommended design target. It mixes I/O, a global cache, and formatting.

# messy_report.py — fixture, not a design target
from __future__ import annotations

import csv
import json
from pathlib import Path
from typing import Any

CACHE: dict[str, int] = {}


def run_report(csv_path: Path, out_dir: Path) -> dict[str, Any]:
    CACHE.clear()
    rows: list[dict[str, Any]] = []
    with csv_path.open(newline="", encoding="utf-8") as handle:
        for row in csv.DictReader(handle):
            sku = row["sku"].strip()
            qty = int(row["qty"])
            CACHE[sku] = CACHE.get(sku, 0) + qty
            rows.append({"sku": sku, "qty": qty})

    total = sum(CACHE.values())
    lines = []
    for sku, qty in sorted(CACHE.items()):
        share = 0 if total == 0 else round(100 * qty / total, 2)
        lines.append(f"{sku}:{qty}:{share}")

    summary = "total={0};skus={1}\n{2}\n".format(
        total, len(CACHE), ";".join(lines)
    )
    payload = {
        "total": total,
        "skus": len(CACHE),
        "cache": dict(CACHE),
        "rows": rows,
    }

    out_dir.mkdir(parents=True, exist_ok=True)
    (out_dir / "summary.txt").write_text(summary, encoding="utf-8")
    (out_dir / "report.json").write_text(
        json.dumps(payload, sort_keys=True, indent=2) + "\n",
        encoding="utf-8",
    )
    print(summary, end="")
    return payload

The fixture writes two output files together. It also prints the summary text. The cache is global and cleared per run.

Duplicate SKUs accumulate inside the CACHE map. Share values use round to two decimals. JSON keys stay sorted for stable bytes.

Artifact: characterization harness

Create a golden directory beside the test file. Store the input CSV and expected outputs. Compare raw bytes, not parsed approximations.

# test_characterize_report.py
from __future__ import annotations

import json
import shutil
from pathlib import Path

import messy_report

FIXTURES = Path(__file__).parent / "goldens" / "report_v1"


def _load_bytes(path: Path) -> bytes:
    return path.read_bytes()


def test_report_matches_golden(tmp_path: Path, capsys):
    src_csv = FIXTURES / "input.csv"
    work_csv = tmp_path / "input.csv"
    shutil.copyfile(src_csv, work_csv)

    messy_report.CACHE["stale"] = 99
    payload = messy_report.run_report(work_csv, tmp_path / "out")

    captured = capsys.readouterr()
    assert captured.out == (FIXTURES / "stdout.txt").read_text(
        encoding="utf-8"
    )
    assert captured.err == ""

    out_dir = tmp_path / "out"
    assert _load_bytes(out_dir / "summary.txt") == _load_bytes(
        FIXTURES / "summary.txt"
    )
    assert _load_bytes(out_dir / "report.json") == _load_bytes(
        FIXTURES / "report.json"
    )

    expected = json.loads(
        (FIXTURES / "report.json").read_text(encoding="utf-8")
    )
    assert payload == expected
    assert messy_report.CACHE == expected["cache"]
    assert "stale" not in messy_report.CACHE

Seed golden files from one known run. Commit those golden files before any extract. Do not edit goldens during the refactor itself.

Use this CSV as goldens/report_v1/input.csv.

sku,qty
A-1,2
B-9,5
A-1,3

Capture stdout from that same CSV once. Keep summary.txt identical to the stdout capture. Keep report.json with sorted keys and a newline.

Seed the oracle with commands

Run these commands against the current fixture. Copy the temporary outputs into goldens. Then rerun pytest until the harness is green.

python - <<'PY'
from pathlib import Path
import messy_report
csv_path = Path("goldens/report_v1/input.csv")
out = Path("/tmp/report_v1")
messy_report.run_report(csv_path, out)
print("wrote", out)
PY
cp /tmp/report_v1/summary.txt goldens/report_v1/stdout.txt
cp /tmp/report_v1/summary.txt goldens/report_v1/summary.txt
cp /tmp/report_v1/report.json goldens/report_v1/report.json
python -m pytest test_characterize_report.py -q

The copy step creates the byte oracle. After this step, tests own the bytes. Later refactor patches must not retouch those files.

Numbered workflow

1. Freeze the entry command

Pick one public function as the boundary. Ignore private helpers for this first pass. Write that single call inside a test.

Record the input path as an explicit Path. The fixture avoids implicit current working directory. That choice keeps goldens portable across machines.

2. Capture goldens once

Run the harness against the current code. Copy outputs into git immediately after. Treat any later mismatch as a regression.

If stdout and summary.txt differ, stop now. The fixture currently prints the same summary. A hidden extra print is a second contract.

3. Add one adversarial input row

Include a duplicate SKU or a zero qty. Do not add ten rows in this step. Extra rows wait until the oracle is stable.

Zero qty still creates a CACHE key today. An empty file fails inside DictReader immediately. Write down which errors are current behavior.

4. Propose extra cases without a rewrite

A second reviewer can suggest extra CSV rows. That reviewer must not rewrite run_report. You accept a row only when goldens stay honest.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option. Point that option at the frozen harness and request extra input rows only.

Reject any suggested patch that rewrites run_report. Reject new tests that drop exact byte compares. The model is a row generator, not an oracle.

5. Change one function

Extract only the summary line formatter. Keep the CSV reading path inside run_report. Keep the cache updates inside run_report for now.

The block below is a proposed extract. Wire it with one assignment. Leave file writes where they already live.

def format_summary(cache: dict[str, int]) -> str:
    total = sum(cache.values())
    lines = []
    for sku, qty in sorted(cache.items()):
        share = 0 if total == 0 else round(100 * qty / total, 2)
        lines.append(f"{sku}:{qty}:{share}")
    return "total={0};skus={1}\n{2}\n".format(
        total, len(cache), ";".join(lines)
    )

Wire it with one assignment in run_report. Run the characterization test after the wire. Stop immediately if the goldens drift.

6. Stop at a green boundary

Do not extract JSON writing in this pass. Do not remove the global cache yet. One safe change is the unit of work.

Check git diff --stat before you commit. Logic files should list two paths at most. A third logic file means you overshot.

Decision table

Observed signal	Characterize first	Rewrite the tests
Duplicate SKUs change totals	Yes	No
You rename a private helper	Yes	No
Product owner changes share formula	No	Yes
File encoding was always wrong	No	Yes, after a bug ticket
You only move writes onto Path	Yes	No
JSON key order was unstable	Yes, after sort_keys	No

If the intended formula is unknown, keep the golden. If the formula is wrong on purpose, file a ticket. Do not hide a known bug inside a rewrite.

When goldens mismatch

Read the pytest diff as a behavior delta. Classify that delta before you touch code. Use this order every time.

1. Restore the input CSV if that file changed.
2. Restore bytes if newline or encoding changed.
3. Restore sort_keys if JSON key order changed.
4. Revert the formatter if share rounding changed.
5. Restore CACHE.clear if the map leaked keys.

Only after those five checks may goldens update. Golden updates need a one-line ticket reason. The word refactor is not a reason.

What the smallest change looks like

The diff should touch formatting logic only. run_report still reads the CSV rows. It still clears CACHE before the accumulation loop.

It still writes summary.txt and report.json. print still uses the same summary string. The return payload still matches report.json.

python -m pytest test_characterize_report.py -q
git diff --stat
git diff -- messy_report.py

Expect one test module to stay green. Expect a small git diff --stat. If three logic files appear, split the work.

Limitations

Characterization tests freeze bugs as well as features. They do not prove full functional correctness. They only prove output stability across edits.

Golden files rot when encoding changes silently. They also rot when JSON separators change. Keep sort_keys=True and explicit trailing newlines.

Global cache tests are order-sensitive across files. Run this module in isolation first. Then add a CACHE reset fixture for the suite.

Floating round to two decimals is a frozen contract. Do not switch rounding modes in the same pass. Pin the current rounding inside the golden file.

This method is slow for interactive GUI state. It is weak for time-dependent report fields. Clock values need a separate pin strategy.

Large binary reports will bloat the git history. Prefer text goldens under a few kilobytes. Hash large blobs instead of committing full copies.

Who should not use this approach

Do not use this on a greenfield module. Write real unit tests for that module instead. There is no legacy contract worth freezing.

Do not characterize a security boundary you will tighten. Do not freeze PII inside golden files. Redact those fields or synthesize the input rows.

Do not treat a generated suggestion as the oracle. The oracle is your committed golden files. Extra rows are proposals until you accept them.

Skip this if you cannot run pytest locally. Remote edits without goldens will drift fast. Byte contracts need a local red-green loop.

Result

You leave the messy repo messy on purpose. The public report stays byte-stable under git. One formatter becomes a testable pure function.

The next extract starts from the same goldens. Repeat the six-step loop without stacking work. Do not batch five extracts into one patch.

If you draft extra CSV rows on a free server, commit goldens first. Review every accepted row by hand before the next extract.