File Counts Stay Green While Walker Extracts Reorder Paths

Directory walkers fail after helper extracts without pinned path lists. Skip reasons and symlink policy must freeze first. Extract one iterator only after those contracts hold.

Messy scanners mix hidden-file rules with ad hoc walk calls. Later helpers reorder paths and swallow cycle errors. Tests that only count files hide those breaks.

The contract to freeze

Pin four observables before any later function move. Relative path order is the first frozen contract. Skip reason strings are the second frozen contract.

Symlink follow flags are the third frozen contract. Duplicate inode handling is the fourth frozen contract. These four pins catch most helper-extract regressions early.

Do not pin mtime values in these tests. Clock noise makes those golden files flake often. Do not pin absolute path prefixes in the golden file. CI hosts differ on temporary directory root prefixes.

Raw os.walk order follows directory readdir results. That order is not stable across host filesystems. Sort rows by POSIX relpath before writing goldens.

Fixture layout

Use a tiny tree, not the real repository. Real trees hide skip-rule bugs in noisy ways. The fixture below is a labeled proposal only.

scan_fix/
  keep.txt
  .secret
  nested/
    inner.txt
    keep_hard.txt   # hardlink to keep.txt when supported
  links/
    to_keep -> ../keep.txt
    loop_a -> loop_b
    loop_b -> loop_a

Build that tree inside a test temp path. Avoid committing broken symlinks on Windows CI runners. Skip symlink cases when the platform lacks them.

Step 1: Record the current walk

Write a recorder that dumps JSON rows only. JSON keeps order and reason codes stable. Treat this recorder as characterization data, not API.

# proposal: characterization recorder, not a public API
from __future__ import annotations

import json
import os
from pathlib import Path
from typing import Any

HIDDEN_PREFIX = "."


def _rel(root: Path, path: Path) -> str:
    return path.relative_to(root).as_posix()


def _inode_key(path: Path) -> tuple[int, int] | None:
    try:
        st = path.lstat()
    except OSError:
        return None
    return (st.st_dev, st.st_ino)


def record_walk(root: Path, follow_symlinks: bool) -> list[dict[str, Any]]:
    rows: list[dict[str, Any]] = []
    seen: set[tuple[int, int]] = set()
    root = root.resolve()

    for dirpath, dirnames, filenames in os.walk(
        root,
        followlinks=follow_symlinks,
        onerror=lambda err: None,
    ):
        dirnames[:] = sorted(dirnames)
        filenames.sort()
        current = Path(dirpath)
        names = list(dirnames) + list(filenames)
        for name in names:
            path = current / name
            skipped = False
            reason = "none"
            kind = "file"
            if name.startswith(HIDDEN_PREFIX):
                skipped = True
                reason = "hidden"
                if path.is_dir() and not path.is_symlink():
                    kind = "dir"
                    dirnames[:] = [item for item in dirnames if item != name]
            elif path.is_symlink():
                kind = "symlink"
                if not follow_symlinks:
                    skipped = True
                    reason = "symlink_unfollowed"
            elif path.is_dir():
                kind = "dir"

            key = _inode_key(path)
            if not skipped and key is not None and key in seen:
                skipped = True
                reason = "duplicate_inode"
            elif key is not None:
                seen.add(key)

            rows.append(
                {
                    "relpath": _rel(root, path),
                    "kind": kind,
                    "skipped": skipped,
                    "reason": reason,
                    "follow_symlinks": follow_symlinks,
                }
            )

    rows.sort(key=lambda row: row["relpath"])
    return rows


def dump_golden(path: Path, rows: list[dict[str, Any]]) -> None:
    text = json.dumps(rows, indent=2, sort_keys=True) + "\n"
    path.write_text(text, encoding="utf-8")

Run the recorder once against the fixture tree. Save that JSON next to the test module. That JSON file becomes the frozen golden contract.

Dump objects with sorted keys, not sorted rows twice. Row order must follow the relpath field only. Double sorting hides accidental reorder bugs much later.

Step 2: Lock skip reasons in tests

Assert full row equality, not loose file counts. File counts stay green while skip reasons drift. A hidden file counted as kept is a miss.

# proposal: characterization tests for the messy walker
import json
import os
import sys
from pathlib import Path

GOLDEN = Path(__file__).parent / "goldens" / "scan_walk.json"


def build_fixture(tmp: Path) -> Path:
    root = tmp / "scan_fix"
    (root / "nested").mkdir(parents=True)
    (root / "links").mkdir()
    (root / "keep.txt").write_text("ok\n", encoding="utf-8")
    (root / ".secret").write_text("nope\n", encoding="utf-8")
    (root / "nested" / "inner.txt").write_text("in\n", encoding="utf-8")
    if sys.platform != "win32":
        os.symlink("../keep.txt", root / "links" / "to_keep")
        os.symlink("loop_b", root / "links" / "loop_a")
        os.symlink("loop_a", root / "links" / "loop_b")
        os.link(root / "keep.txt", root / "nested" / "keep_hard.txt")
    return root


def test_walk_matches_golden(tmp_path: Path) -> None:
    root = build_fixture(tmp_path)
    rows = record_walk(root, follow_symlinks=False)
    expected = json.loads(GOLDEN.read_text(encoding="utf-8"))
    assert rows == expected

Keep the golden JSON file in git. Review those golden rows like production code changes. Wrong reasons in goldens freeze the wrong behavior.

Step 3: Cover cycles and hardlinks

Add symlink cycles only when follow_symlinks stays false. Python os.walk can loop when followlinks is true. Characterization should record the safe default path first.

Hardlinks share device and inode numbers here. The second path should record the duplicate_inode reason. Use lstat so symlinks are not followed.

Platforms without os.link should skip extra rows. Do not split goldens unless CI requires it. One POSIX golden keeps the contract quite readable.

python -m pytest tests/test_scan_walk_char.py -q

Refresh goldens only through an explicit local command. Do not refresh them inside failing CI jobs. Accidental refreshes hide extract regressions.

# proposal: explicit golden refresh, never implicit
if os.environ.get("UPDATE_SCAN_GOLDEN") == "1":
    dump_golden(GOLDEN, record_walk(root, follow_symlinks=False))

Step 4: Extract one iterator

Move record_walk into scan.py without any edits. Keep the JSON comparison as the only gate. Do not rename reason strings during the move.

# proposal: smallest extract after the golden is green
from scan import record_walk as scan_entries

The production caller should consume scan_entries after that. Delete duplicate walk loops after the assertion passes. That function move is the smallest safe change.

The rglob trap

Path.rglob looks like a cleaner walker extract. It is not equivalent to the current os.walk loop. Order, directory rows, and symlink handling all change.

Path.rglob yields paths as it discovers them. Directory entries may disappear from the result set. Hidden prefixes stay in the stream unless filtered later.

Do not swap rglob in during the extract. Swap only after a new golden exists. New goldens are a behavior change, not a move.

Step 5: Recheck the golden JSON

Run the same pytest node after the move. Diff the JSON file against git HEAD. Any path reorder means the extract has failed.

python -m pytest tests/test_scan_walk_char.py -q
git diff -- tests/goldens/scan_walk.json

Empty diff is the pass condition here. Recount tests are not enough for this refactor. Reason strings must match byte for byte.

Decision table

Change	Pin first	Extract now	Wait
Hidden prefix skip	reason=hidden	Yes	No
Stable path order	Sorted POSIX relpath	Yes	No
Unfollowed symlink	reason=symlink_unfollowed	Yes	No
Hardlink duplicate	reason=duplicate_inode	Yes	No
Follow symlinks	Cycle and escape policy	No	Visited realpath set
Sort by mtime	None	No	Drop that sort
Glob ignore file	Match list golden	No	Separate extract

Read the table before accepting a patch. Follow-symlink work waits for a visited set. mtime sorting should be dropped, not extracted now.

Glob ignore files need their own golden list. Do not fold glob parsing into this extract. One iterator is the whole allowed change.

Using a remote model after pins

After the golden JSON is committed, a model can propose the extract. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option.

Feed the messy walker and the pinned tests only. Reject any patch that changes skip reasons or path order. Local pytest remains the source of truth here.

Limitations

Characterization records current bugs as frozen truth too. Hidden-file rules may be too blunt here. Duplicate inode skips may drop wanted names.

Golden equality does not prove race safety yet. Walkers can miss files created during the scan. This method does not test permission errors deeply.

Do not use this walker on untrusted trees. Symlink following can escape the intended root path. Followlinks true still risks infinite symlink cycles here.

Network filesystems can reuse inode numbers across mounts. Those hosts need a different duplicate identity key. Device plus inode is not globally unique there.

Who should skip this method

Skip this method without a JSON-capable test runner. Skip it when walk order is not a contract. Skip it for one-off scripts on trusted files.

Skip it if hidden-file policy is still debated. Freeze the policy in a ticket first. Then record goldens after that policy lands.

Freeze skip reasons and path order first. Extract one walker after the golden matches. Add cycle guards only as a later change.