Freeze the Output Directory, Then Extract One Seam

Messy packages leak behavior through files, not return values.
A fixture-tree oracle pins that leak before any extract.
The smallest safe change then lives behind one seam.

Untested directories rarely fail inside a single helper.
They fail when a path, encoding, or filename shifts.
Function-level tables miss these package-boundary file effects.

Why interior oracles are not enough

Return-shape checks help after you already have purity.
Most messy repos do not start with a pure function.
They start with a script that writes a folder.

A green helper test can still ship a broken tree.
The report file can change name while helpers pass.
Operators observe the directory listing, not the boolean.

This workflow treats the output folder as the contract.
You freeze relative paths and file bytes first.
Only then do you extract one interior seam.

Freeze this boundary, not that interior

Pin three observables before you touch source.

1. The entrypoint command or public function call.
2. The input fixture directory, including nested files.
3. Output paths, SHA-256 hashes, and selected text.

Leave these unfrozen until the extract is done.

1. Local helper names and private call graphs.
2. Comment text, import order, and lint noise.
3. Temporary files that the process later deletes.

Decision table

Signal	Freeze now	Change now
Entrypoint argv	Yes	No
Input fixture tree	Yes	No
Output relative paths	Yes	No
Output file hashes	Yes	No
Private helper name	No	Yes, after a green oracle
Inline constant	No	Yes, if hashes stay equal
New required flag	No	No, that is a contract change

Treat a contract change as separate product work.
Do not mix it with a structural extract.

Example package (proposal, not production)

The sample below is a compact stand-in package.
Do not treat it as measured production telemetry.
It exists so the harness stays copyable.

# billing/report.py
from __future__ import annotations

import json
from pathlib import Path


def run_report(src: Path, dest: Path) -> None:
    dest.mkdir(parents=True, exist_ok=True)
    total = 0
    lines = []
    for raw in sorted(src.glob("*.json")):
        payload = json.loads(raw.read_text(encoding="utf-8"))
        amount = int(payload.get("amount") or 0)
        sku = str(payload.get("sku") or "unknown")
        if amount < 0:
            amount = 0
        total += amount
        lines.append(f"{sku},{amount}")
    (dest / "lines.csv").write_text("\n".join(lines) + "\n", encoding="utf-8")
    (dest / "summary.txt").write_text(f"total={total}\n", encoding="utf-8")
    (dest / "ok.flag").write_text("1\n", encoding="utf-8")

The module writes three files and clamps negative amounts.
There is no test and no documented contract.
A rewrite can rename files and still look tidy.

Step 1 — Capture the entrypoint

Keep one public call. Do not fan out through internals.

# billing/__main__.py
from pathlib import Path
from billing.report import run_report

if __name__ == "__main__":
    run_report(Path("in"), Path("out"))

Record the exact call you will freeze.

python -m billing

If the real repo uses a CLI parser, freeze those flags too.
Do not freeze a private function you plan to delete.
The oracle should survive a later helper rename.

Step 2 — Build a committed fixture tree

Put known inputs under version control.
Keep them small, explicit, and slightly ugly.

fixtures/report/in/a.json
fixtures/report/in/b.json
fixtures/report/in/neg.json

{"sku": "A-1", "amount": 10}

{"sku": "B-9", "amount": 5}

{"sku": "Z-0", "amount": -3}

Ugly cases belong in the tree on purpose.
Missing keys, negatives, and sort order leak into files.
A tidy fixture set will under-pin the real contract.

Add one extra file only when it changes an output byte.
Duplicate happy-path invoices do not strengthen the oracle.
Coverage here means distinct directory effects, not line hits.

Step 3 — Record the output tree once

Run the entrypoint against a temp copy of fixtures.
Store relative paths plus SHA-256 hashes.

# tools/characterize_report.py
from __future__ import annotations

import hashlib
import json
import shutil
import subprocess
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]
FIX_IN = ROOT / "fixtures" / "report" / "in"
GOLDEN = ROOT / "fixtures" / "report" / "golden.json"


def hash_tree(folder: Path) -> dict[str, str]:
    out: dict[str, str] = {}
    for path in sorted(folder.rglob("*")):
        if path.is_file():
            rel = path.relative_to(folder).as_posix()
            digest = hashlib.sha256(path.read_bytes()).hexdigest()
            out[rel] = digest
    return out


def run_into(dest: Path) -> dict[str, str]:
    if dest.exists():
        shutil.rmtree(dest)
    dest.mkdir(parents=True)
    work_in = dest / "in"
    work_out = dest / "out"
    shutil.copytree(FIX_IN, work_in)
    subprocess.run(
        [sys.executable, "-m", "billing"],
        cwd=dest,
        check=True,
    )
    return hash_tree(work_out)


if __name__ == "__main__":
    recorded = run_into(ROOT / ".tmp-characterize")
    GOLDEN.write_text(json.dumps(recorded, indent=2) + "\n", encoding="utf-8")
    print(json.dumps(recorded, indent=2))

Commit golden.json only after you inspect the files.
Hashes without a manual read will pin accidental junk.
Open each output path and confirm the bytes look intentional.

python tools/characterize_report.py
find .tmp-characterize/out -type f -print | sort
cat .tmp-characterize/out/lines.csv
cat .tmp-characterize/out/summary.txt

If a file contains a clock, normalize it before hashing.
Byte-exact oracles fail on timestamps by design.
Strip volatile fields in the recorder, not in production code.

VOLATILE_PREFIXES = ("generated_at=", "run_id=")


def normalize_bytes(raw: bytes) -> bytes:
    text = raw.decode("utf-8")
    kept = []
    for line in text.splitlines(keepends=True):
        if line.startswith(VOLATILE_PREFIXES):
            continue
        kept.append(line)
    return "".join(kept).encode("utf-8")

Apply normalize_bytes only to known text reports.
Do not silently drop binary files from the tree.
Missing paths are contract breaks, not noise.

Step 4 — Turn the recording into a failing-closed test

The test must fail when a filename or byte changes.

# tests/test_report_tree.py
from __future__ import annotations

import json
from pathlib import Path

from tools.characterize_report import GOLDEN, ROOT, run_into


def test_output_tree_matches_golden() -> None:
    expected = json.loads(GOLDEN.read_text(encoding="utf-8"))
    actual = run_into(ROOT / ".tmp-test-report")
    assert actual == expected

Run it twice before any edit.
A flaky oracle is not an oracle.

python -m pytest tests/test_report_tree.py -q
python -m pytest tests/test_report_tree.py -q

If the second run drifts, stop the refactor.
Find the source of nondeterminism first.
Sort orders, dict iteration, and clocks are the usual leaks.

Step 5 — Diagnose a hash mismatch before coding

When hashes differ, do not guess from the pytest traceback.
Rebuild both trees and diff the paths.

python - <<'PY'
import json
from pathlib import Path
exp = json.loads(Path("fixtures/report/golden.json").read_text())
# paste or load the actual dict from the failing assertion
PY
diff -ru .tmp-characterize/out .tmp-test-report/out

Classify the mismatch into one bucket.

1. Extra path: the extract wrote a new file.
2. Missing path: the extract stopped writing a file.
3. Same path, new hash: bytes changed inside the file.
4. Renamed path: the contract moved, even if content matches.

Renames are not refactors under this oracle.
Restore the old relative path before any further extract.

Step 6 — Insert one seam, change nothing else

A seam is a replaceable function with the same bytes out.
Extract the line formatter. Do not touch paths or filenames.

def format_line(sku: str, amount: int) -> str:
    return f"{sku},{amount}"

Call it from run_report. Keep the three output names identical.
Re-run the tree test. Hashes must match exactly.

If hashes drift, revert the extract.
The seam is wrong even if the helper looks cleaner.
Interior style never outranks the directory contract.

One seam means one new function, not a new package layout.
Do not move files, split modules, or add config objects yet.
Those steps change import paths and hide extra diffs.

Step 7 — Make the smallest interior change

Only after the seam is green, change the helper body.
Keep the package boundary frozen.

Allowed now:

1. Rename locals inside format_line.
2. Replace the f-string with a constant template.
3. Add a type alias that does not leak to files.

Forbidden now:

1. New output files.
2. CSV header rows.
3. Different sort order.
4. UTF-16 or extra trailing spaces.

Re-run the same test after each tiny edit.
Stop at the first hash mismatch.
Revert that edit before starting another one.

Three commits, not one heroic diff

Split the work so bisect can name the break.

1. Commit the fixture tree, recorder, and golden hashes.
2. Commit the seam extract with no helper-body change.
3. Commit the interior helper edit only.

If commit three fails CI, revert only that commit.
The oracle and the seam should remain green.
Mixing all three hides which step changed the tree.

Where a coding model belongs

Do not ask a model to invent the fixture tree.
Humans choose inputs because they know the ugly cases.

After golden.json is committed, a model may draft the seam.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode offers free model access and a free server option.
Those can propose the extract against the frozen tree.
They do not replace the hash comparison.

Feed the model the golden listing and the current module.
Ask for one seam only. Reject extra files and new flags.
If the tree test fails, discard the draft. Do not negotiate hashes.

A useful prompt states the frozen paths in literal form.
It also states the files that must not appear.
Vague cleanup requests produce contract changes dressed as refactors.

Limitations

This oracle is byte-exact and therefore brittle on purpose.
Timestamps, time zones, and random IDs will break it.
Strip those fields or normalize them before hashing.

It also assumes a serial, disk-based contract.
Network calls, clocks, and shared databases need other oracles.
Do not hash a directory that contains secrets.

Large binary trees make reviews expensive.
Prefer text fixtures until the contract is obvious.
If the product is the binary layout, hash plus a documented viewer.

The harness also will not catch missing features.
It pins current behavior, including current bugs.
Fixing a clamped negative amount is a contract change.
Schedule that fix after the seam extract, in a fourth commit.

Who should not use this approach

Skip this workflow in four cases.

1. You must change filenames or output schema now.
2. The process is concurrent and the tree is racy.
3. Outputs embed credentials, tokens, or personal data.
4. You already have a stable public API with tests.

In those cases, freeze a different boundary.
Do not pretend a fixture tree is universal.
A typed API test is a better oracle for library packages.

Skip it for exploratory spikes with no callers.
Characterization cost only pays when the tree has users.
Throwaway scripts can be rewritten without a golden folder.

What smallest safe change means here

Smallest means one seam and one helper body.
Safe means the committed tree hashes stay identical.
Anything else is a product change, not a refactor.

Keep the fixture tree in source control.
Let the hashes, not the diff, accept the extract.
If a free model draft names the seam, the tree still judges it.