Ignore Walkers Drift Until Tests Freeze Sorted Path Sets

Messy ignore walkers fail without a frozen match set. Characterization tests must pin those match sets first. Extract one include predicate only after those pins hold.

Repo scanners mix globs, ignore files, and directory walks. Teams extract helpers to shorten the messy loop. The helper often changes symlink policy by accident.

Green suites then hide extra vendor library files. Backup copies leak into packaging without a red test. Build artifacts pollute later hash inputs as well.

Observables that actually matter

Pin three values. Drop everything else from the contract.

1. Sorted relative paths returned by the walker.
2. Whether directory symlinks are followed at all.
3. The first matching ignore pattern for each path.

Skip mtime values. Skip inode numbers on every platform. Skip raw os.walk order on mixed filesystems. Those numbers are not the product contract.

Fixture the walker must see

Build a temporary tree inside the test. Do not reuse the live application repo.

src/app.py
src/app.py.bak
src/nested/mod.py
build/out.bin
vendor/lib.py
.gitignore
link_src -> src

Put two ignore lines into .gitignore.

*.bak
build/

This tree stays small on purpose. Each rule still has one clear victim. The symlink sits beside src, not inside it.

Step 1 — Record the match set

Call the current walker once. Sort POSIX-style relative paths. Assert the full list, not a substring.

# test_repo_walker_contract.py
from pathlib import Path
import os
import tempfile
import unittest

from repo_walker import collect_paths  # existing messy entry point


def make_fixture(root: Path) -> None:
    (root / "src" / "nested").mkdir(parents=True)
    (root / "build").mkdir()
    (root / "vendor").mkdir()
    (root / "src" / "app.py").write_text("print(1)\n", encoding="utf-8")
    (root / "src" / "app.py.bak").write_text("old\n", encoding="utf-8")
    (root / "src" / "nested" / "mod.py").write_text("x=1\n", encoding="utf-8")
    (root / "build" / "out.bin").write_bytes(b"\x00\x01")
    (root / "vendor" / "lib.py").write_text("v=1\n", encoding="utf-8")
    (root / ".gitignore").write_text("*.bak\nbuild/\n", encoding="utf-8")
    os.symlink(root / "src", root / "link_src", target_is_directory=True)


def relpaths(root: Path, paths) -> list[str]:
    return sorted(Path(p).relative_to(root).as_posix() for p in paths)

Label the list below as current behavior. It is not a design award.

NOFOLLOW_SET = [
    ".gitignore",
    "src/app.py",
    "src/nested/mod.py",
    "vendor/lib.py",
]

.bak files stay out. build/ stays out. vendor/lib.py stays in until product rules change.

Step 2 — Record symlink policy

Run the same fixture twice. Flip only follow_symlinks.

class WalkerContractTest(unittest.TestCase):
    def test_nofollow_match_set(self):
        with tempfile.TemporaryDirectory() as tmp:
            root = Path(tmp)
            make_fixture(root)
            got = collect_paths(root, follow_symlinks=False)
            self.assertEqual(relpaths(root, got), NOFOLLOW_SET)

    def test_follow_does_not_invent_bak_files(self):
        with tempfile.TemporaryDirectory() as tmp:
            root = Path(tmp)
            make_fixture(root)
            got = collect_paths(root, follow_symlinks=True)
            rel = relpaths(root, got)
            self.assertIn("src/app.py", rel)
            self.assertNotIn("src/app.py.bak", rel)
            self.assertNotIn("build/out.bin", rel)

Do not assert symlink target identity here. Assert membership of logical paths. Duplicate src/... entries through link_src/ are a separate pin.

Add that pin only after you observe it once.

    def test_follow_link_src_prefix_is_explicit(self):
        with tempfile.TemporaryDirectory() as tmp:
            root = Path(tmp)
            make_fixture(root)
            got = collect_paths(root, follow_symlinks=True)
            rel = relpaths(root, got)
            linked = [p for p in rel if p.startswith("link_src/")]
            # Characterization: freeze today's count, do not guess.
            self.assertEqual(len(linked), len(linked))
            self.assertTrue(all(not p.endswith(".bak") for p in linked))

Replace the tautology on len(linked) after the first local run. Write the observed integer into the assertion. Leave a comment that the integer is recorded, not desired.

Step 3 — Record the first matching rule

Path sets hide rule-order bugs. Two patterns can skip the same file. Later extracts then swap which pattern won.

RULES = ("*.bak", "build/")


def first_rule(rel: str) -> str | None:
    name = Path(rel).name
    parts = Path(rel).parts
    if name.endswith(".bak") or name == "*.bak":
        if Path(rel).suffix == ".bak":
            return "*.bak"
    if parts and parts[0] == "build":
        return "build/"
    return None

That helper is test-side only. Do not import it from production yet. Table the outcomes beside the match set.

Relative path	First rule	In nofollow set
src/app.py	none	yes
src/app.py.bak	*.bak	no
src/nested/mod.py	none	yes
build/out.bin	build/	no
vendor/lib.py	none	yes
.gitignore	none	yes
link_src/app.py	none	nofollow: absent

Keep this table in the test module docstring. Update the table and the list together. A mismatch means the extract already drifted.

Step 4 — Extract one predicate

Stop after one function. Do not rewrite os.walk in the same patch.

Proposed production shape, not an executed patch:

# proposed: repo_walker.py
from pathlib import Path
from fnmatch import fnmatch


def should_include(rel_posix: str, rules: tuple[str, ...] = ("*.bak", "build/")) -> bool:
    path = Path(rel_posix)
    parts = path.parts
    for rule in rules:
        if rule.endswith("/") and parts and parts[0] == rule.rstrip("/"):
            return False
        if fnmatch(path.name, rule):
            return False
    return True

Wire it from the existing loop. Leave directory traversal untouched. Re-run the three pins after the call site changes.

python -m unittest test_repo_walker_contract.py -v

If NOFOLLOW_SET moves, revert the extract. Do not edit the expected list to match the helper. The list is the contract. The helper is the suspect.

Step 5 — Keep generation off the contract

Drafting a predicate is cheap after pins exist. The pins stay local. A remote draft cannot replace the fixture tree.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option. Those two facts are the only product claims used here. No model names, quotas, or hardware details belong in this workflow.

Use a local run as the gate. Paste failing assertion text into a draft prompt only after the fixture exists. Compare the draft predicate against NOFOLLOW_SET and the rule table. Reject drafts that follow link_src when the pin says nofollow.

Command checklist

Run these in order. Do not skip the dry fixture listing.

1. Create the temp tree through make_fixture.
2. Print relpaths once with follow disabled.
3. Print relpaths once with follow enabled.
4. Fill NOFOLLOW_SET from step 2 output.
5. Fill the linked-prefix count from step 3 output.
6. Extract should_include only.
7. Re-run unittest without editing expects.

python - <<'PY'
from pathlib import Path
import tempfile
from test_repo_walker_contract import make_fixture, relpaths
from repo_walker import collect_paths

with tempfile.TemporaryDirectory() as tmp:
    root = Path(tmp)
    make_fixture(root)
    print("NOFOLLOW")
    print("\n".join(relpaths(root, collect_paths(root, follow_symlinks=False))))
    print("FOLLOW")
    print("\n".join(relpaths(root, collect_paths(root, follow_symlinks=True))))
PY

Capture both blocks in the test file. Future diffs then show walker drift, not formatting noise.

Failure analysis

Three extract failures show up in this fixture. Name them before the patch lands.

1. fnmatch on the full relative string, not path.name.
2. build matched as a substring inside rebuild/ names.
3. Directory symlinks followed because Path.rglob was swapped in.

Failure 1 reintroduces src/app.py.bak when the pattern is written as */*.bak. Failure 2 is absent from this tree, so add rebuild/out.bin if you ship prefix rules. Failure 3 changes len(linked) without touching NOFOLLOW_SET.

Assert all three. One list is not enough coverage.

Limitations

This method records behavior. It does not prove the ignore language is complete. Nested .gitignore files are out of scope. Negation rules such as !keep.bak are out of scope. Character-class globs are out of scope.

Windows symlink creation may require extra process privileges. Skip test_follow_* on platforms where os.symlink raises. Do not rewrite policy in that skip branch. Mark the skip as an environment limit.

fnmatch is not gitignore. Trailing-slash directory rules need the parts[0] check shown above. Encoding is pinned to UTF-8 text files in the fixture. Binary names are not covered.

Who should not use this approach

Do not use this flow for a greenfield scanner. Write the real gitignore grammar first in that case. Do not use it when legal review forbids fixture copies of production trees. Synthetic files above are enough for rule pins.

Do not extract traversal, rule parsing, and symlink policy together. That patch is not the smallest safe change. Do not feed the live monorepo to a remote draft. Path names can be sensitive even without file bytes.

Skip the method if no test can create TemporaryDirectory. Characterization without a fixture is just log reading. Log reading does not freeze match sets.

What the pins refuse to bless

A shorter loop is not a safer loop. Sorted path sets are the product surface for this walker. Rule order is the second surface. Symlink follow flags are the third.

After those three hold, one predicate extract is measurable. Without them, the helper is a style change with hidden scan drift. Keep the list, the table, and the follow flag in one module. Change product intent by editing those pins first, then the code.

If the pins already run locally, a free-server draft of the predicate can be compared against them. Leave the fixture on the machine when the tree cannot travel.