Pin Relpaths, Sizes, and SHA-256 Before One Walker Extract

The safe refactor order is not a style debate. Pin a content-addressed file inventory before any extract. Extract one walker function only after that pin.

Mixed path walkers hide platform bugs in plain sight. Native separators and mtimes make snapshots flake often. A rewrite without those pins remains a guess.

The problem in one inventory

A messy repo often ships two collectors together. One collector uses os.walk with raw string joins. Another uses Path.rglob with narrow suffix filters.

Hidden files and symlink policy then diverge in silence. Tests pass on one laptop and fail later. The inventory is the missing contract for this refactor.

Duplicate collection looks busy and still changes nothing. A second rglob for txt files may add zero paths. Only a frozen inventory can prove that overlap.

Decision table for the pin

Record three fields for every regular file. Use a relative POSIX path from a fixed root. Store the integer size and a hex SHA-256.

Drop absolute paths because they encode the host. Drop native separators because they encode the OS. Drop mtime, mode, inode, and device identifiers.

Field	Pin	Why
posix relpath	yes	Stable file identity
size	yes	Fast mismatch signal
sha256	yes	Content identity
mtime	no	Changes on copy
mode	no	Follows umask
symlink target	policy	Decide before extract

Symlink policy belongs in a written module note. Follow, skip, or error on each link. Do not leave that choice implicit in the walker.

Why size and hash travel together

Size mismatches are cheap to print in CI logs. Hash mismatches catch silent byte edits later. Together they separate rename drift from content drift.

Rename without content change shows missing plus extra. Same digest on two relpaths means a copy. The assertion does not infer renames for you.

That limitation is acceptable for a walker extract. You want empty diffs, not a migration log. Empty missing, extra, and changed is the merge gate.

Messy collector to pin

The module below is the characterization target. It mixes string joins with Path.rglob. Treat every branch as observed behavior, not intent.

# messy_walk.py — characterization target, not a design
from pathlib import Path
import os


def legacy_collect(root):
    out = []
    root_s = str(root)
    for dirpath, _, files in os.walk(root_s):
        for name in files:
            out.append(os.path.join(dirpath, name))
    for path in Path(root_s).rglob("*.txt"):
        text = str(path)
        if text not in out:
            out.append(text)
    return [Path(item) for item in out]

Membership uses raw strings from two APIs. That check is case-sensitive and separator-sensitive. It is also the first flake to pin.

Artifact: a characterization harness

The harness below is a labeled proposal. Treat it as unexecuted until you run it. Place it beside the messy walker, not inside.

# inventory_char.py
from __future__ import annotations

import hashlib
from pathlib import Path
from typing import Iterable

Inventory = list[tuple[str, int, str]]


def posix_relpath(root: Path, file_path: Path) -> str:
    rel = file_path.resolve().relative_to(root.resolve())
    return rel.as_posix()


def sha256_file(path: Path, chunk: int = 64 * 1024) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        while True:
            block = handle.read(chunk)
            if not block:
                break
            digest.update(block)
    return digest.hexdigest()


def inventory_from_paths(root: Path, files: Iterable[Path]) -> Inventory:
    rows: Inventory = []
    seen: set[str] = set()
    for file_path in files:
        path = Path(file_path)
        if path.is_symlink() or not path.is_file():
            continue
        rel = posix_relpath(root, path)
        if rel in seen:
            continue
        seen.add(rel)
        rows.append((rel, path.stat().st_size, sha256_file(path)))
    rows.sort(key=lambda row: row[0])
    return rows


def assert_inventory_equal(expected: Inventory, actual: Inventory) -> None:
    if expected == actual:
        return
    expected_map = {row[0]: row for row in expected}
    actual_map = {row[0]: row for row in actual}
    missing = sorted(set(expected_map) - set(actual_map))
    extra = sorted(set(actual_map) - set(expected_map))
    changed = [
        key
        for key in sorted(set(expected_map) & set(actual_map))
        if expected_map[key] != actual_map[key]
    ]
    raise AssertionError(
        f"missing={missing!r} extra={extra!r} changed={changed!r}"
    )


def dump_inventory(rows: Inventory) -> None:
    for rel, size, digest in rows:
        print(f"{rel}\t{size}\t{digest}")

The sort key is the POSIX relpath only. Equal trees then compare as plain lists. The assertion names missing, extra, and changed paths.

Skip symlinks in the inventory builder on purpose. The messy collector may still return them as paths. Decide follow versus skip before the extract patch.

Deduping by POSIX relpath is part of the contract. Two string forms of one file become one row. That is how separator leaks stop reaching the golden list.

Fixture tree you can replay

Build a tiny fixture with known bytes. Avoid relying on live repository clutter for this pin. Keep names that stress separators and sort order.

# test_inventory_char.py
from pathlib import Path

from inventory_char import assert_inventory_equal, inventory_from_paths
from messy_walk import legacy_collect

FIXTURE_SPEC = {
    "a/readme.txt": b"alpha\n",
    "a/B/data.bin": b"\x00\x01\x02",
    ".hidden": b"dot\n",
    "z-last/file.txt": b"omega",
}


def write_fixture(root: Path) -> None:
    for rel, data in FIXTURE_SPEC.items():
        path = root.joinpath(*rel.split("/"))
        path.parent.mkdir(parents=True, exist_ok=True)
        path.write_bytes(data)

This fixture stays deterministic across copies and hosts. Sizes and hashes do not depend on clock. Hidden files are included on purpose in the spec.

Compute expected rows from the legacy collector once. Print the TSV dump and store it beside the test. Do not hand-type digests from memory.

Numbered workflow

Follow this order without skipping any pin.

1. Copy the messy walker into a test helper. Do not edit behavior in that copy. Name it legacy_collect(root) and return list[Path].
2. Write the fixture into a temporary directory. Use tmp_path from pytest for isolation. Keep the root argument explicit in every call.
3. Run inventory_from_paths on legacy_collect output now. Freeze that list as the expected rows. Commit those rows next to the characterization test.
4. Run the same inventory on a second operating system. Confirm POSIX relpaths still match after the copy. Separator leaks show up as extra or missing keys.
5. Extract one function named collect_files(root). Keep skip rules identical to the legacy helper. Do not fold extra cleanup into that patch.
6. Re-run the inventory assertion on both collectors. Missing, extra, and changed must stay empty lists. Only then delete the legacy_collect alias.

def test_extract_keeps_inventory(tmp_path: Path) -> None:
    write_fixture(tmp_path)
    expected = inventory_from_paths(tmp_path, legacy_collect(tmp_path))
    actual = inventory_from_paths(tmp_path, collect_files(tmp_path))
    assert_inventory_equal(expected, actual)

The test does not inspect walker internals at all. It inspects the inventory contract and nothing else. That restriction is the method, not a style choice.

python -m pytest test_inventory_char.py -q --tb=short

Smallest safe change

Replace join style inside collect_files and nothing else. Do not fold ignore rules into the same patch. Do not add recursion limits in that same patch.

Here is a typical extract sketch for later review. Treat it as unlabeled until the inventory stays green. Sorting dirnames only stabilizes walk order for humans.

from pathlib import Path
import os


def collect_files(root: Path) -> list[Path]:
    root = Path(root).resolve()
    found: list[Path] = []
    for dirpath, dirnames, filenames in os.walk(root, followlinks=False):
        dirnames.sort()
        filenames.sort()
        for name in filenames:
            path = Path(dirpath) / name
            if path.is_symlink():
                continue
            if path.is_file():
                found.append(path)
    return found

Walker order can remain an implementation detail after the pin. The inventory sort remains the published contract for reviewers. Read missing, extra, and changed before any style comment.

If the txt rglob adds no extra inventory rows, drop it. The pin is the proof, not the nearby comment. Dead collectors survive for years without that proof.

Where a free coding model fits

A model helps only after the inventory pin exists. It does not replace the fixture or the assertion.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. Paste the inventory test and the legacy walker into that workspace. Ask for the smallest collect_files extract that keeps the assertion green.

Do not ask the model to invent skip rules. Do not ask it to strip hidden files. Feed the decision table as hard constraints.

Review the patch as a stranger would review it. Re-run the inventory test on your machine. Discard any change that touches ignore policy.

Diagnosing a red inventory

Read the AssertionError fields in this numbered order.

1. missing means the extract dropped paths unexpectedly. Restore skip rules from legacy_collect before adding globs. Do not compensate with new patterns yet.
2. extra means the extract gained paths unexpectedly. Check followlinks and hidden-file policy next. Check whether rglob used to filter suffixes.
3. changed means path identity held and content drifted. Confirm the fixture write is still deterministic today. Confirm no rewriter touched file bytes.

Print a TSV dump when the inventory list is large. Use relpath, size, and digest columns only. Diff that dump like any other golden file.

A rename without a byte change splits across two fields. You will see one missing path and one extra path. Do not merge those rows by digest during an extract.

Limitations

This harness does not prove text encoding policy at all. It hashes bytes and never decodes strings. Equal bytes with different intended encodings still match.

It does not classify symlink cycles for you automatically. followlinks=False is a local documented choice. Put that choice in the test module docstring.

Large binaries will dominate hash runtime in CI. Multi-gigabyte assets do not belong in this pin. Exclude those globs in the legacy helper first.

Permission errors are not inventory tuple fields here. A walker that throws needs a separate characterization test. Capture exception type and path, not file bytes.

Case-folding filesystems can collapse two POSIX relpaths. The golden list then depends on the host. Run the pin on the OS that production uses.

Who should not use this approach

Do not treat this pin as a security audit. Content hashes do not detect secret leakage. They only detect logical tree drift after a change.

Skip this workflow for live upload directories entirely. Concurrent writes make the inventory race under you. Snapshot a frozen fixture instead of the live tree.

Skip it if you cannot run pytest locally first. A suggested extract without a local rerun is a guess. The pin only lives in your test runner.

Teams that need inode-preserving copies should pin more fields. This contract is logical tree identity only. It is not backup fidelity and not forensic imaging.

Close

Pin relpaths, sizes, and SHA-256 before the extract. Keep ignore policy out of that first patch. Let missing, extra, and changed drive the review.

Style debates can wait until the pin stays green. Dead rglob calls should fall only after that proof. The inventory is the contract, not the walker shape.