Freeze the Visited Path Set and followlinks Before Extracting a Walker

Do not extract a directory walker from messy scripts first.
The current visited path set is the contract.
Freeze that set, then extract one function only.

Messy scanners hide walk policy inside nested loops.
Symlink flags, ignore dirs, and suffix filters drift together.
A cleaner helper often changes which files CI processes.

Measure the set, not the style

A walker is a set producer, not a naming exercise.
Two helpers can look identical and visit different paths.
The delta appears as extra lint or missing fixtures.

Four silent deltas show up in real refactors.

1. Default followlinks=False versus a glob that follows dirs.
2. Skipping dot directories in only one branch.
3. Applying ignore rules after the full descent.
4. Mixing Path.resolve() with raw relative names.

Those deltas need a snapshot, not a benchmark.
Do not argue from taste. Compare path lists instead.

Freeze four facts

Record live behavior as committed JSON.
Do not record intent. Do not record hopes.

1. Sorted relative paths the current code visits.
2. The followlinks value actually passed to os.walk.
3. Directory names pruned through dirnames[:].
4. Whether downstream code depends on walk order.

If consumers hash files, order can stay free.
If consumers concatenate in walk order, freeze order too.
Write that choice down before any extract.

1. Build a deterministic fixture

Do not characterize a dirty worktree first.
Local __pycache__ and .venv will flake the snapshot.
Build a tiny fixture that encodes the messy cases.

# fixture_tree.py — labeled example layout, not a production dump
from pathlib import Path
import os


def build_fixture(root: Path) -> None:
    src = root / "src"
    src.mkdir(parents=True)
    (src / "app.py").write_text("print(1)\n", encoding="utf-8")
    (src / ".hidden.py").write_text("print(0)\n", encoding="utf-8")
    nested = src / "pkg"
    nested.mkdir()
    (nested / "mod.py").write_text("x = 1\n", encoding="utf-8")

    (root / "build").mkdir()
    (root / "build" / "out.py").write_text("# generated\n", encoding="utf-8")
    (root / "vendor").mkdir()
    (root / "vendor" / "lib.py").write_text("# third_party\n", encoding="utf-8")

    linked = root / "linked_src"
    if not linked.exists():
        try:
            os.symlink(src, linked, target_is_directory=True)
        except OSError:
            pass

The tree stays small on purpose.
It encodes hidden files, generated dirs, vendor, and a directory symlink.
File symlinks are the wrong test for followlinks.

os.walk(..., followlinks=False) still lists file symlinks.
It refuses to descend into directory symlinks.
That distinction belongs beside the snapshot, in comments.

2. Instrument the current walker

Do not rewrite the scanner in this step.
Wrap it. Dump a stable record.

# characterize_walk.py
from __future__ import annotations

from pathlib import Path
import hashlib
import json
import os

IGNORE_DIRS = {"build", "vendor", ".git", "__pycache__"}
EXTENSIONS = {".py"}


def messy_collect(root: Path, *, followlinks: bool = False) -> list[str]:
    found: list[str] = []
    for dirpath, dirnames, filenames in os.walk(root, followlinks=followlinks):
        dirnames[:] = [
            d for d in dirnames
            if d not in IGNORE_DIRS and not d.startswith(".")
        ]
        for name in filenames:
            if name.startswith("."):
                continue
            path = Path(dirpath) / name
            if path.suffix not in EXTENSIONS:
                continue
            rel = path.relative_to(root).as_posix()
            found.append(rel)
    return sorted(found)


def fingerprint(paths: list[str]) -> str:
    blob = "\n".join(paths).encode("utf-8")
    return hashlib.sha256(blob).hexdigest()


def dump_snapshot(root: Path, out: Path, *, followlinks: bool = False) -> None:
    paths = messy_collect(root, followlinks=followlinks)
    payload = {
        "followlinks": followlinks,
        "ignore_dirs": sorted(IGNORE_DIRS),
        "extensions": sorted(EXTENSIONS),
        "paths": paths,
        "sha256": fingerprint(paths),
    }
    out.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")

Run the dumper once against the fixture.
Commit walk_snapshot.json next to the test.
Keep the JSON boring so reviewers can read it.

Expected fixture paths with followlinks=False are narrow.
src/app.py and src/pkg/mod.py should appear.
src/.hidden.py stays out. build/out.py stays out.
linked_src/app.py stays out because descent is skipped.

Why dirnames[:] is part of the contract

Mutating dirnames in place is the prune API for os.walk.
Building a new list without assigning back does not prune.
That bug visits extra trees and fails the snapshot.

Prune during descent, not after collecting every path.
Post-filtering still walks vendor and build.
On a large repo that cost shows up as CI time, not as a type error.

Ignore rules that run after Path.rglob are a different walker.
Do not treat them as equivalent in the first extract.
Record which algorithm you actually run.

3. Lock the snapshot in tests

The test must fail on set drift.
Helper names are irrelevant in this layer.

# test_walk_snapshot.py
import json
from pathlib import Path

from characterize_walk import dump_snapshot, messy_collect
from fixture_tree import build_fixture

SNAPSHOT = Path("walk_snapshot.json")


def test_visited_paths_match_committed_snapshot(tmp_path):
    repo = tmp_path / "repo"
    build_fixture(repo)
    live_path = tmp_path / "live.json"
    dump_snapshot(repo, live_path, followlinks=False)
    live = json.loads(live_path.read_text(encoding="utf-8"))
    committed = json.loads(SNAPSHOT.read_text(encoding="utf-8"))
    assert live["followlinks"] is False
    assert committed["followlinks"] is False
    assert live["paths"] == committed["paths"]
    assert live["sha256"] == committed["sha256"]


def test_directory_symlink_is_not_descended(tmp_path):
    repo = tmp_path / "repo"
    build_fixture(repo)
    paths = messy_collect(repo, followlinks=False)
    assert "src/app.py" in paths
    assert "src/pkg/mod.py" in paths
    assert all(not p.startswith("linked_src/") for p in paths)


def test_followlinks_true_is_a_different_contract(tmp_path):
    repo = tmp_path / "repo"
    build_fixture(repo)
    if not (repo / "linked_src").exists():
        return
    off = set(messy_collect(repo, followlinks=False))
    on = set(messy_collect(repo, followlinks=True))
    assert on >= off
    assert any(p.startswith("linked_src/") for p in on)

Commands stay short and local.

python -c "from pathlib import Path; from fixture_tree import build_fixture; from characterize_walk import dump_snapshot; r=Path('fixture_repo'); build_fixture(r); dump_snapshot(r, Path('walk_snapshot.json'))"
pytest test_walk_snapshot.py -q

Green tests mean the set is known.
They do not mean the code is clean.
Clean comes after the set is pinned.

4. Extract one function, same flags

Move messy_collect in a single diff.
Do not change followlinks.
Do not replace dirnames[:] filtering.

Keep ignore application during descent.
Filtering after a full walk can visit huge trees.
It can also follow a directory symlink you meant to skip.

Do not switch to Path.rglob in the same commit.
rglob is another walker, not a rename.
Path.walk exists since Python 3.12. Treat that move as change two.

followlinks=True can introduce cycles in bad trees.
Do not enable it because a model prefers symmetry.
If the snapshot says false, keep it false.

5. Optional model pass after the gate exists

A coding model can draft the extract.
It cannot invent the visited set for you.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free model access and a free server option.
Use either only after walk_snapshot.json is committed.
Send the messy function plus the snapshot, then keep diffs that preserve paths and followlinks.
Drop any patch that changes the SHA-256.

That single comparison is the whole review rule.
No extra product steps are required after that.

Decision table

Current signal	Freeze in snapshot	Safe first extract	Same-diff hazard
followlinks omitted	explicit False	keep os.walk	switch to rglob
dirnames[:] prune	ignore_dirs list	keep in-walk prune	filter after full walk
dot-name skip	skip_dot: true	keep startswith(".")	glob **/*.py only
suffix check	extensions list	keep path.suffix	add MIME sniffing
consumers sort later	order free	return sorted list	freeze raw walk order as API
consumers concat in order	order frozen	return walk order	sort for readability

What this snapshot cannot catch

The JSON does not freeze file bytes.
A walker extract should not read files yet.
Split readers into a later diff.

It does not freeze stat times or inode numbers.
Most scanners do not need those fields.
Hash them only if a consumer already does.

It does not prove Windows junction behavior.
The fixture stores POSIX relative paths.
Re-run the dumper on the OS you ship.

Nondeterministic trees break the contract.
Unignored temp files will churn sha256.
Add those names to IGNORE_DIRS first.

Who should not use this

Do not use this if the walker deletes files.
Characterization must stay read-only.
Add a dry-run flag before any snapshot.

Do not snapshot a generated site tree as input.
Build outputs change by design.
Pin the source set the builder consumes.

Do not use this as a type-check substitute.
The snapshot pins behavior, not signatures.
Keep argument names explicit on extract.

Skip the model pass if you lack pytest.
Unreviewed extracts still drift the set.
The JSON is the gate, not the chat log.

Limitations

The snapshot is only as strong as the fixture.
A two-file tree misses nested build/ dirs.
Add one nested generated directory when production has them.

os.walk error handling is a separate contract.
The default ignores some scandir errors.
Do not add onerror in the extract diff.

Order-sensitive consumers need a different snapshot.
Sorting in the dumper can hide a real bug.
If order matters, store the unsorted walk list.

Symlink creation can fail without extra privileges.
Then linked_src is absent and one test returns early.
That skip is honest. It is not coverage.

Close

The smallest safe walker refactor preserves the path set.
Name followlinks. Name the ignore list.
Commit the visited paths. Then move one function.

The SHA-256 is the merge gate for that extract.
Helper names can wait until the set is stable.