Sort the Path Set Before You Extract an Ignore Matcher

Do not extract ignore helpers from a messy scanner yet. Filesystem walk order is not a stable contract. Freeze the sorted path set first, then extract one matcher.

The failure this sequence prevents

A god function walks the tree and filters names. Engineers then extract a should_ignore helper by eye.

The extract often changes slash handling or substring matches. Tests still pass because they assert length only.

Length checks hide dropped files and extra files. Sorted tuple equality does not hide those misses. Ignore hits also need their own frozen table.

A compact messy scanner

The next module is a proposed example only. It is not claimed as production scanner code. It mixes walk order, substring ignores, and relative paths.

# scan_repo.py — proposed example, not executed production code
from __future__ import annotations

import os
from pathlib import Path
from typing import Iterable

IGNORE_SUBSTR = [".git", "__pycache__", "node_modules", ".tmp"]


def scan_repo(root: str) -> list[str]:
    found: list[str] = []
    root_path = Path(root)
    for dirpath, dirnames, filenames in os.walk(root):
        dirnames[:] = [d for d in dirnames if not _bad(d)]
        for name in filenames:
            if _bad(name):
                continue
            full = Path(dirpath) / name
            rel = os.path.relpath(full, root_path)
            found.append(rel.replace("\\", "/"))
    return found


def _bad(name: str) -> bool:
    lowered = name.lower()
    for token in IGNORE_SUBSTR:
        if token in lowered:
            return True
    return False

This filter is substring based, not path based. A file named notes.tmp.md is dropped.

The token .git also drops .gitkeep files. os.walk order still leaks into the return list.

What you must pin

Pin observables that a refactor must not change. Skip internals that the extract will replace.

Observable	Pin method	Why it matters
Returned paths	sorted tuple of relative POSIX paths	Walk order is not stable
Ignored paths	sorted tuple of all_files - kept	Pruned dirs hide extra drops
Ignore hits	sorted tuple of (relpath, token)	Substring rules drop keep files
Slash form	no backslashes in returned paths	Windows fixtures change joins

Do not pin dirnames mutation as an API contract. Pin the user-visible path set as the contract instead. Record ignore hits so silent drops stay explainable.

Workflow

Follow this order on a dirty fixture. Do not skip the gold recording step.

1. Build a dirty fixture tree on local disk.
2. Record the current path set as gold.
3. Record ignore hits with the matched token.
4. Assert sorted tuples, uniqueness, and POSIX slash form.
5. Extract one matcher function and nothing else.
6. Re-run the same frozen assertions without gold edits.

Stop if any gold tuple changes during the extract. That shift is a behavior change, not a refactor.

Step 1: Build a dirty fixture

Create paths that stress substring rules on purpose. Include a near-miss name beside a true hit. Include nested ignored directories and keep-style files.

# test_scan_repo.py — proposed characterization harness
from __future__ import annotations

import os
from pathlib import Path

from scan_repo import scan_repo

TOKENS = [".git", "__pycache__", "node_modules", ".tmp"]


def make_fixture(tmp_path: Path) -> Path:
    layout = [
        "README.md",
        "src/app.py",
        "src/bigit.log",
        "src/.gitkeep",
        ".git/config",
        "build/.tmp/out.bin",
        "node_modules/pkg/index.js",
        "tests/__pycache__/t.pyc",
        "docs/notes.tmp.md",
    ]
    for rel in layout:
        path = tmp_path / rel
        path.parent.mkdir(parents=True, exist_ok=True)
        path.write_text("x\n", encoding="utf-8")
    return tmp_path


def iter_layout(root: Path) -> list[str]:
    rels: list[str] = []
    for dirpath, _, filenames in os.walk(root):
        for name in filenames:
            full = Path(dirpath) / name
            rel = os.path.relpath(full, root).replace("\\", "/")
            rels.append(rel)
    return sorted(rels)

src/bigit.log does not contain the token .git. The keep file src/.gitkeep does contain that token. docs/notes.tmp.md contains .tmp inside the filename.

Step 2: Freeze the path set

Call the scanner once against the dirty fixture. Sort the result before any equality comparison. Store gold in the test, not in comments.

def test_scan_repo_path_set_is_stable(tmp_path: Path) -> None:
    root = make_fixture(tmp_path)
    got = tuple(sorted(scan_repo(str(root))))
    gold = (
        "README.md",
        "src/app.py",
        "src/bigit.log",
    )
    assert got == gold

This gold is a proposal until the first run. Run the test before you edit the tuple. Three surviving files is expected with these tokens.

.git/config never appears because .git is pruned. node_modules children never appear for the same reason. notes.tmp.md disappears because the filename matches .tmp.

Step 3: Freeze ignore hits

Path-set tests do not explain vanished files well. Add a hit table for every layout path. Walk without pruning so ignored children still show up.

def test_ignore_hits_match_current_tokens(tmp_path: Path) -> None:
    root = make_fixture(tmp_path)
    hits: list[tuple[str, str]] = []
    for rel in iter_layout(root):
        matched = ""
        for part in rel.split("/"):
            for token in TOKENS:
                if token in part.lower():
                    matched = token
                    break
            if matched:
                break
        if matched:
            hits.append((rel, matched))
    gold_hits = (
        (".git/config", ".git"),
        ("build/.tmp/out.bin", ".tmp"),
        ("docs/notes.tmp.md", ".tmp"),
        ("node_modules/pkg/index.js", "node_modules"),
        ("src/.gitkeep", ".git"),
        ("tests/__pycache__/t.pyc", "__pycache__"),
    )
    assert tuple(hits) == gold_hits

The hit table will surprise most first-time reviewers. A .gitkeep file matches .git as a substring. Keep that behavior until a later named change.

Step 4: Pin slash form and uniqueness

Return order is not a safe contract. Duplicates appear after sloppy relative path joins. POSIX slashes must hold on Windows fixtures too.

def test_scan_repo_paths_are_posix_and_unique(tmp_path: Path) -> None:
    root = make_fixture(tmp_path)
    raw = scan_repo(str(root))
    assert all("\\" not in p for p in raw)
    assert all(not p.startswith("/") for p in raw)
    assert len(raw) == len(set(raw))

Combine this test with the sorted gold tuple. Together they catch extras, missing files, and duplicate paths. Order-only churn no longer looks like a failure.

Step 5: Extract one matcher

Change only the predicate used by the walk. Keep the scan_repo control flow exactly intact. Keep _bad as a thin wrapper for old tests.

def is_ignored(name: str, tokens: Iterable[str]) -> bool:
    lowered = name.lower()
    return any(token in lowered for token in tokens)


def _bad(name: str) -> bool:
    return is_ignored(name, IGNORE_SUBSTR)

Re-run both gold tests after this extract. .gitkeep must still drop under the old token. If bigit.log disappears, stop and revert the patch.

Step 6: Keep the next change separate

Git-style path rules are a feature change. Do not fold them into the matcher extract. Ship the extract alone with unchanged gold tuples.

Open a second patch for real ignore rules later. Update gold in that second patch only. Review that second diff as a behavior change.

Optional help drafting gold tuples

Drafting hit tables is tedious and easy to mistype. A coding model can propose tuples from a fixture dump.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option. Draft gold tuples there, then rerun pytest against the real fixture.

Treat every generated gold tuple as untrusted input. The harness still has to match on disk. Reject any extract that moves the path set.

Do not ask a model to clean ignore rules now. Cleaning is a behavior change, not a refactor. The harness exists to make that change visible.

Limitations

This method does not prove the matcher is correct. It proves the extract did not move current behavior. Substring ignores remain wrong for names like .gitkeep.

It does not pin symlink policy by default. Add a symlink to the fixture if callers care. os.walk skips directory symlinks unless followlinks is set.

Gold tuples rot when the fixture layout changes. Keep fixture layout construction in one test helper. Review gold diffs with the same care as API diffs.

Empty directories never appear in this file scanner. A later extract that lists dirs will move gold. That is another reason to keep extracts small.

Who should not use this approach

Do not use this sequence for a greenfield scanner. Write explicit ignore rules first in that case. Do not extract while also renaming return types.

Do not skip the hit table when files vanish. Length assertions are not enough for ignore logic. Do not mix Windows path tests with unpinned slash form.

Security reviews still need a separate human pass. An ignore matcher can hide files or leak them. Characterization tests only freeze what the messy code already did.

Commands

Run the harness inside a clean virtualenv. Do not mix this run with unrelated pytest noise.

python -m venv .venv
. .venv/bin/activate
pip install pytest
pytest test_scan_repo.py -q

After the extract, run the same command again. Diff only scan_repo.py inside that extract patch. The test file should stay unchanged during a true refactor.

If gold must change, you are not refactoring. Split the work and name the behavior change. Keep the matcher extract boring and easy to review.

Close

Freeze the sorted path set and the ignore hits. Extract one predicate after those tuples stay green. Feature changes belong in a later patch with new gold.