Skip Rules Drift Until You Freeze Path Fixtures

Skip rules drift until you freeze path fixtures.
Do not extract a filter from mixed walker code.
Pin include and exclude sets with tests first.
Then change only one function after that pin.

The mixed-job failure

Repo scanners often mix three separate jobs together.
They walk disks, apply skips, and emit paths.
Those jobs share separators, prefixes, and sort keys.

Hidden files look skipped on one machine.
A backslash path then leaks into output.

A trailing slash breaks a prefix check.
Extracting should_skip without fixtures hides all three.

Substring skips also create silent false positives.
A folder named node_modules_backup can vanish entirely.

That vanishing looks like a clean tree walk.
It is actually a silent matching bug in disguise.

Freeze four observables

Freeze four observables before any source edit.

1. Sorted relative paths that remain included.
2. Sorted relative paths that were skipped.
3. Skip reason strings keyed by relative path.
4. Separator style stored in those keys.

Relative paths beat absolute paths in these tests.
Absolute paths encode local machine layout details here.
That layout is not the behavior under test.

Sort both lists with a stable ASCII order.
Do not trust os.walk order across platforms.
Two machines can visit siblings in different order.

Teaching fixture layout

The layout below is a teaching example only.
It is not a claim about a production repo.

scan_root/
  src/app.py
  src/node_modules_backup/keep.py
  node_modules/pkg/index.js
  .env
  .github/workflows/ci.yml
  build/out.bin
  Build/out.bin
  docs/guide.md

That tree hits four common skip bugs at once.
Substring match, hidden files, case, and walk drops.

Messy walker to characterize

Label this module as unexecuted teaching code only.
Copy it into a scratch folder before running tests.

# messy_walker.py — teaching example, not production
from __future__ import annotations

import os
from typing import Dict, List, Tuple

SKIP_REASONS: Dict[str, str] = {}


def scan_tree(root: str) -> List[str]:
    included: List[str] = []
    SKIP_REASONS.clear()
    for dirpath, dirnames, filenames in os.walk(root):
        # In-place dirnames mutation changes which files appear.
        dirnames[:] = [
            d for d in dirnames
            if not _raw_skip(os.path.join(dirpath, d), root)[0]
        ]
        for name in filenames:
            full = os.path.join(dirpath, name)
            skipped, reason = _raw_skip(full, root)
            rel = full[len(root):].lstrip("\\/")
            if skipped:
                SKIP_REASONS[rel] = reason
                continue
            included.append(rel)
    return included


def _raw_skip(full: str, root: str) -> Tuple[bool, str]:
    rel = full[len(root):].lstrip("\\/")
    # Bug: substring match, not path segment match.
    if "node_modules" in rel:
        return True, "substring:node_modules"
    # Bug: hidden check uses the raw relative string.
    if rel.startswith("."):
        return True, "hidden-prefix"
    # Bug: prefix check is case-sensitive and separator-naive.
    if rel.startswith("build/") or rel.startswith("build\\"):
        return True, "prefix:build"
    return False, ""

The slice on full paths is already fragile.
Missing trailing separators produce a bad relative key.

os.walk plus in-place dirnames mutation drops whole subtrees.
Those drops never appear in the skip map.

Characterization tests

These tests pin current behavior, including present bugs.
They do not define the ideal skip language yet.

# test_messy_walker.py — teaching example
from __future__ import annotations

from pathlib import Path

from messy_walker import SKIP_REASONS, scan_tree


def _make_tree(base: Path) -> None:
    files = [
        "src/app.py",
        "src/node_modules_backup/keep.py",
        "node_modules/pkg/index.js",
        ".env",
        ".github/workflows/ci.yml",
        "build/out.bin",
        "Build/out.bin",
        "docs/guide.md",
    ]
    for rel in files:
        path = base / rel
        path.parent.mkdir(parents=True, exist_ok=True)
        path.write_text("x\n", encoding="utf-8")


def test_included_paths_are_frozen(tmp_path: Path) -> None:
    _make_tree(tmp_path)
    included = sorted(scan_tree(str(tmp_path)))
    assert included == sorted([
        "src/app.py",
        "Build/out.bin",
        "docs/guide.md",
    ])


def test_skipped_keys_and_reasons_are_frozen(tmp_path: Path) -> None:
    _make_tree(tmp_path)
    scan_tree(str(tmp_path))
    reasons = dict(sorted(SKIP_REASONS.items()))
    assert reasons["src/node_modules_backup/keep.py"] == "substring:node_modules"
    assert reasons[".env"] == "hidden-prefix"
    assert "build/out.bin" in reasons or "build\\out.bin" in reasons

Run one command after the files exist locally.

python -m pytest test_messy_walker.py -q

A failing assertion is the whole point here.
It names drift before any extract lands in git.

Note that Build/out.bin stays included today.
The prefix check is case-sensitive in this walker.

.github/workflows/ci.yml never reaches the skip map.
The walk dropped .github during dirnames mutation.

src/node_modules_backup/keep.py is skipped by substring.
That is wrong product behavior, not desired policy.
It is still current behavior in this module.
Characterization tests must pin that fact first.

Why reasons belong in the pin

A boolean skip bit hides why a path vanished.
Two bugs can then cancel in the included list.

A hidden-prefix drop and a walk drop look identical.
Reason strings split those failure modes apart.

Keep reason strings stable during the extract.
Do not reword them for readability in that diff.

Rewording is a second behavioral change then.
It needs its own assertion update later.

Decision table for the extract

Use this table when a helper draft appears.
Every last-column no blocks the first extract.

Relative path	Current skip	Reason pinned	Safe to change now
src/app.py	no	none	no
src/node_modules_backup/keep.py	yes	substring:node_modules	no
node_modules/pkg/index.js	yes	substring or walk drop	no
.env	yes	hidden-prefix	no
.github/workflows/ci.yml	yes via dir drop	missing map entry	no
build/out.bin	yes	prefix:build	no
Build/out.bin	no	case gap	no
docs/guide.md	no	none	no

A later change can flip one row on purpose.
That later change needs a new explicit test.

Numbered workflow

Follow these steps in order, without skipping.
Do not skip the characterization pin at all.

1. Copy the messy walker into a scratch module.
2. Build the smallest fixture tree you can.
3. Record included paths as a sorted list.
4. Record skip keys and their reason strings.
5. Commit tests that assert both frozen records.
6. Ask a model for one helper extract only.
7. Reject any extra cleanup in that diff.
8. Re-run the same pytest command after the extract.
9. Keep the extract only if both records match.

Step six is optional tooling, not the method.
The characterization pin is not optional in this method.

Smallest safe extract

This extract is a labeled proposal only.
Apply it only after the tests above pass.

def should_skip(rel: str) -> tuple[bool, str]:
    # Preserve current bugs. Do not fix matching yet.
    if "node_modules" in rel:
        return True, "substring:node_modules"
    if rel.startswith("."):
        return True, "hidden-prefix"
    if rel.startswith("build/") or rel.startswith("build\\"):
        return True, "prefix:build"
    return False, ""

Wire scan_tree to call should_skip next.
Leave dirnames mutation untouched in this first diff.
Leave the root-length slice untouched too.
Those are later changes with their own pins.

If the extract rewrites substring matching, reject it.
If it starts emitting POSIX-only keys, reject it.
If it stops mutating dirnames, walk output changes.
That change is not the smallest safe one.

Do not sort inside the first extract

Sorting output looks harmless and even professional here.
It still changes callers that rely on walk order.
Characterization tests used a sorted copy only.

The production function can stay unsorted for now.
Keep that distinction in the extract review.

How to read a failed pin

A shorter included list usually means extra skips.
A longer included list means a skip disappeared.

A reason mismatch means the helper reworded a label.
A missing skip key means walk mutation dropped a directory.

Do not edit the test to match the extract.
Fix the extract to match the test instead.
That rule is the whole method here.

Free model drafts still need the pin

A coding model can draft the helper quickly.
It cannot see skip drift without fixtures.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access can propose that helper.
The free server option can host the same drafting session.
Neither claim includes named models or measured quotas here.
Treat both as availability notes, not as a benchmark.

Keep the extract prompt narrow on purpose.

Extract should_skip(rel) -> (bool, str).
Do not change skip semantics.
Do not normalize case.
Do not alter os.walk mutation.
Keep current reason strings.

Paste the frozen tests next to that prompt.
If the draft flips any decision-table row, discard it.
Your pytest output remains the only merge gate.

If you already draft extracts in MonkeyCode, attach these fixtures first.

Limitations

This method locks present bugs into tests on purpose.
That lock is useful for a first extract.
It is harmful if you needed a spec change now.

It does not cover symlink loops at all.
It does not cover permission errors on walk.
It does not cover Unicode normalization of names.
It does not cover gitwildmatch ignore-file grammar.

Substring reasons are not a public skip API.
Do not export them to users after the extract.
They exist to detect drift, not to document product rules.

Platform separators still leak into stored keys.
A Linux fixture may not match a Windows key.
Pin separator style or run tests on both.

Who should not use this approach

Skip this method on greenfield scanners entirely.
Write a real skip spec and test that spec.

Skip it when product intent must change today.
Do not freeze a false positive you must ship now.

Skip it when skip rules are a security boundary.
Path traversal needs a specified matcher, not a snapshot.

Skip it when the walker has no test runner yet.
The pin is a test, not a comment in chat.

What to change after the extract

Only then pick one semantic bug to fix.
Add one new row that states the desired skip.
Update should_skip for that row alone next.
Re-run the original characterization tests after that.

If old tests fail, you changed too much.
Split the work into a later change.

The core conclusion stays the same throughout.
Skip rules drift until you freeze path fixtures.
Extract after the freeze, not before that freeze.