Pin Walk Order and Byte Totals Before One Relpath Extract

Do not extract a path helper from a messy walker first. Freeze sorted relative paths and integer byte totals before any cut. Glob order, symlink follows, and float rounding change reports without a traceback.

Silent drift, not a crash

os.walk follows disk order, not your mental model. Two clones can list the same files in different sequences. JSON dumps then shuffle object keys unless you sort them.

A size total stored as float drifts across platforms. One extra symlink visit doubles a counted directory. Characterization must pin those three channels before a cleanup commit.

Exit-code-only tests miss this class of bug. The process still returns zero. The report JSON just permutes, and reviewers treat the diff as noise.

A tangled reporter worth pinning

The module below is a labeled proposal, not production history. It mixes walking, formatting, and I/O in one function. That mix is the refactor target, not a claimed private codebase.

# proposal: messy inventory reporter (not a live service)
from __future__ import annotations

import json
import os
from pathlib import Path
from typing import Any


def dump_inventory(root: str, out_path: str) -> dict[str, Any]:
    report: dict[str, Any] = {"root": root, "entries": [], "bytes": 0.0}
    for dirpath, dirnames, filenames in os.walk(root, followlinks=True):
        dirnames.sort()
        for name in filenames:
            full = os.path.join(dirpath, name)
            try:
                st = os.stat(full, follow_symlinks=True)
            except OSError as exc:
                report["entries"].append({"path": full, "error": str(exc)})
                continue
            report["entries"].append(
                {
                    "path": full,
                    "size": float(st.st_size),
                    "mtime": st.st_mtime,
                }
            )
            report["bytes"] += float(st.st_size)
    Path(out_path).write_text(json.dumps(report, indent=2), encoding="utf-8")
    return report

The function writes a file and returns a nested dict. Tests that only assert a zero exit miss reorder bugs. Pin the dict shape after canonicalization, not the raw print stream.

Absolute paths embed the temp root into every golden. Float sizes and raw st_mtime values embed host noise. followlinks=True can re-enter the same tree through a symlink.

What to pin, and what to drop

Pin four observables and ignore the rest on purpose. Relative paths must be POSIX-style and lexicographically sorted. The walker must set followlinks to false unless a test says otherwise.

Byte totals must be integers, never floats from os.stat. JSON must use sort_keys and separators that you freeze. Skip inode, device, and owner fields; they are host noise.

Skip mtime unless a downstream consumer prints timestamps. If you must print times, convert to UTC and drop subseconds. Locale-dependent date strings are not a stable golden.

Unicode names need an explicit normalization form. NFC and NFD can look identical in a terminal. They are not equal as golden strings.

Freeze protocol

1. Isolate the reporter behind a function that accepts a root path. Keep stdout unused during characterization. Return a dict the test can canonicalize.
2. Build a temp fixture tree with files, a gap directory, and a symlink. Include one broken symlink and one nested empty folder. Keep the fixture under 20 files so diffs stay readable.
3. Run the reporter twice against that tree in one process. Do not rebuild the tree between the two calls. A pure read path must be stable inside one run.
4. Canonicalize both outputs with the same dump helper. Compare the encoded text, not Python object identity. Hash the text if the payload grows later.
5. Fail the test if the two canonical payloads differ. Intra-process drift means the walker is already unsafe. Do not extract helpers until that check is green.
6. Write the canonical payload to a committed golden file. Store it next to the test, not in /tmp. Reviewers then see inventory changes as test diffs.
7. Re-run on a second machine or container before you extract. Linux and macOS can disagree on symlink stat results. Windows path separators will fail an unnormalized golden.
8. Only then extract one relpath helper and keep the walker intact. Do not sort, filter, or change followlinks in the same commit. One behavior change per diff.

# proposal: characterization harness
import hashlib
import json
import os
import tempfile
import unittest
from pathlib import Path

GOLDEN = Path(__file__).with_name("inventory.golden.json")


def posix_rel(root: str, full: str) -> str:
    rel = os.path.relpath(full, root)
    return rel.replace("\\", "/")


def canonicalize(report: dict) -> str:
    entries = []
    for item in report["entries"]:
        path = item["path"]
        if path.startswith(report["root"]):
            path = posix_rel(report["root"], path)
        entries.append(
            {
                "path": path,
                "size": int(item.get("size", 0)),
                "error": item.get("error"),
            }
        )
    entries.sort(key=lambda row: row["path"])
    payload = {
        "entries": entries,
        "bytes": int(report["bytes"]),
        "count": len(entries),
    }
    return json.dumps(payload, sort_keys=True, separators=(",", ":"))


def build_tree(root: Path) -> None:
    (root / "a").mkdir()
    (root / "a" / "one.txt").write_bytes(b"aaaa")
    (root / "b").mkdir()
    (root / "b" / "two.txt").write_bytes(b"bb")
    (root / "b" / "gap").mkdir()
    (root / "link_to_a").symlink_to(root / "a", target_is_directory=True)
    (root / "broken").symlink_to(root / "missing.txt")


class InventoryGoldens(unittest.TestCase):
    def test_intra_process_stability_and_golden(self) -> None:
        with tempfile.TemporaryDirectory() as td:
            root = Path(td)
            build_tree(root)
            from inventory_report import dump_inventory  # labeled proposal import

            first = canonicalize(dump_inventory(str(root), str(root / "out1.json")))
            second = canonicalize(dump_inventory(str(root), str(root / "out2.json")))
            self.assertEqual(first, second)
            if not GOLDEN.exists():
                GOLDEN.write_text(first + "\n", encoding="utf-8")
                self.fail("wrote inventory.golden.json; re-run to pin")
            expected = GOLDEN.read_text(encoding="utf-8").strip()
            self.assertEqual(first, expected)
            digest = hashlib.sha256(first.encode("utf-8")).hexdigest()
            self.assertEqual(len(digest), 64)

The first run may write the golden and fail on purpose. That is a recording step, not a product bug. Commit the file only after you inspect every path and byte total.

Decision table

Signal	Pin in golden	Skip	Reason
Relative POSIX paths, sorted	Yes		Walk order otherwise shuffles JSON arrays
Integer byte totals	Yes		Float addition is not a stable identity
followlinks=False	Yes		Symlink loops inflate counts
Broken symlink error string class	Yes		Missing targets must stay visible
Raw st_mtime		Skip	Host clocks and FS precision differ
Inode / device ids		Skip	Values are machine-specific
Absolute temp roots		Skip	CI workspaces never match laptops
Pretty-printed JSON whitespace		Skip	Canonical dump already freezes separators

Use the table when a reviewer asks to "just snapshot stdout." Stdout still contains host paths. The table names the exact fields that survive a machine change.

The only allowed change

Extract _relpath(root, path) -> str and nothing else. Leave os.walk, os.stat, and JSON writing in the original function. The helper must not follow symlinks and must not call stat.

# proposal: smallest extract after goldens pass
def _relpath(root: str, full: str) -> str:
    rel = os.path.relpath(full, root)
    return rel.replace("\\", "/")

Wire the helper only at the dict-building site. Do not "improve" error messages in the same patch. Do not switch followlinks to false in that commit if the golden still encodes the old visits.

If the golden still includes symlink-followed duplicates, split the work. First commit: pin current follow behavior. Second commit: set followlinks=False and update the golden in isolation. Third commit: extract _relpath.

Commands that keep the pin honest

python -m unittest test_inventory_goldens -v
python -c "import json,pathlib; p=pathlib.Path('inventory.golden.json'); print(p.read_text()[:200])"
git diff -- testdata/inventory.golden.json

Run the unittest module, not a custom wrapper, on the first pass. Print a prefix of the golden so you see POSIX slashes. Inspect git diff on the golden before you squash.

Add a second command that hashes the golden in CI. A hash mismatch is cheaper to read than a huge JSON fail. Keep the full JSON for humans, the hash for the gate.

python -c "import hashlib,pathlib; b=pathlib.Path('inventory.golden.json').read_bytes(); print(hashlib.sha256(b).hexdigest())"

Failure analysis

Windows backslashes break an unnormalized path golden. NFC versus NFD names break macOS checkouts of the same repo. Trailing slashes on root change relpath output for the root itself.

Concurrent writers in the fixture directory race the walker. Network filesystems can return stale stat sizes. Short-lived temp files from antivirus scanners appear as extra entries.

Broken symlinks must be present in the fixture. If you omit them, the extract can swallow OSError later. Empty directories never appear in filenames; pin that absence explicitly in comments, not in fake rows.

Who should not use this approach

Skip this protocol on a greenfield package with pure functions. You already control order and types. Characterization goldens would only freeze accidental design.

Skip it when the walker is a secret scanner. Inventory snapshots can copy credential filenames into git. Redact or exclude those trees before recording.

Skip it when no test runner exists yet. A committed golden without an automated compare will rot. Do not extract helpers on reviewer memory alone.

Skip it for one-off scripts you will delete the same day. The freeze cost exceeds the edit cost. Use the protocol on modules that will outlive the current ticket.

Limitations

This artifact does not prove semantic correctness of the report. It only proves the next edit did not change pinned bytes. A wrong total that is stable will stay wrong.

It does not replace property tests for path traversal. A helper that accepts .. can still escape root. Add a dedicated unit test for that case after the extract.

It does not pin permission bits or owner names. Those fields need a dedicated fixture user, which most CI images lack. Leave them out of the golden.

Intra-process stability does not prove cross-process stability. Environment variables, working directory, and umask can still leak. Re-run the golden under a clean cwd in CI.

After the golden exists

A coding assistant is optional once the fixture fails locally. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can draft the _relpath extract against that fixture. The committed golden remains the acceptance check, not the model output.

Reject any patch that retouches walk order and path math together. Reject any patch that rewrites error strings to chase a prettier golden. Keep the reporter boring until the helper has its own tests.

Pin walk order and integer byte totals first. Extract one relpath helper second. Treat every extra cleanup as a later ticket with its own golden diff.