Pin a Path Inventory Before You Extract One Ignore Predicate

Do not extract ignore logic from a messy walker first.
Freeze every keep-or-drop decision on a fixture tree.
Then change one predicate and leave globbing untouched.

A mixed walker usually glob-matches, ignores, and classifies together.
That mix hides the public keep-or-drop contract from tests.
The first extract then changes file counts without failing.

The failure this harness targets

Many repos grow a scan module that does three jobs.
It expands globs, drops ignored paths, and labels survivors.
A later refactor splits those jobs without a freeze.

The usual breakage is silent count drift in CI.
Nested drops vanish and dot-directories leak into results.
Tests on the new matcher miss the old loop order.

This article pins the public inventory, not private helpers.
The inventory is a sorted JSON list of path decisions.
A SHA-256 digest makes later drift obvious in CI.

What the inventory must freeze

Record four fields for every visited path.
Use relative path, node kind, keep flag, and reason code.
Do not record mtimes, inodes, or absolute prefixes.

Reason codes stay small and stable across patches.
Use only dot, pattern, extension, or keep.
Do not store free-text messages. They churn every edit.

The digest covers canonical JSON bytes only.
Encode UTF-8, sort keys, and drop extra whitespace.
Use POSIX separators and exactly one trailing newline.

Reason order is part of that contract.
A path can match both a dot rule and a name list.
Swapping if branches keeps flags equal and still fails.

Build a checked-in fixture tree

Do not scan the live working tree for this test.
Live trees contain caches, editor files, and secrets.
A fixture under tests/fixtures/inventory_tree is the input.

Create the tree from a committed manifest.
The builder must be idempotent on every run.
Re-running it must not change hashes by itself.

# tests/fixtures/manifest.py
# Proposed fixture. Not a gitignore engine.

TREE = [
    "README.md",
    "src/app.py",
    "src/app.pyc",
    "src/.hidden.py",
    "src/vendor/lib.py",
    "build/out.js",
    "docs/guide.md",
    "docs/.cache/tmp.md",
    ".git/HEAD",
    "nested/dir/.env",
    "nested/dir/ok.txt",
    "assets/logo.PNG",
    "assets/logo.png",
]

That set covers dots, vendor, build, case, and extensions.
Add rows when a production bug appears later.
Never delete a row. Change its keep flag instead.

Numbered workflow

1. Isolate the messy walker behind one function

Wrap the current scanner in a single callable.
Use inventory(root) -> list[dict] as the surface.
Do not change behavior in this wrapping step.

Return one dict per visited path.
Keys are path, kind, keep, and reason.
Paths are POSIX relative strings with no leading slash.

2. Canonicalize and hash the inventory

Sort rows by path, then by kind.
Dump JSON with sort_keys=True and tight separators.
Hash the UTF-8 bytes with SHA-256.

Print the hex digest beside the row counts.
Store the expected digest next to the fixture.
A mismatch fails before any extract lands.

3. Treat the digest as the merge gate

The extract may rewrite private helpers only.
The digest must not move after that rewrite.
If it moves, revert and shrink the change.

4. Extract only is_ignored

Move pattern matching into one pure function.
Keep globbing and classification in the walker.
Do not extract two concerns in one patch.

5. Re-run the inventory harness

The digest must match the frozen value exactly.
Spot-check two dropped paths and two kept paths.
Stop there. Do not clean neighboring functions yet.

Runnable characterization harness

The code below is a complete local example.
It is not production ignore logic.
Treat it as a labeled proposal and rename freely.

Ignore rules start inlined inside the walk loop.
That inlined form is the messy baseline to freeze.
The later extract only lifts the predicate.

"""characterization_inventory.py

Freeze keep-or-drop decisions before extracting is_ignored.
Run: python characterization_inventory.py
"""
from __future__ import annotations

import hashlib
import json
import os
import tempfile
from pathlib import Path

IGNORE_DIRS = {".git", "build", "vendor"}
IGNORE_SUFFIXES = {".pyc"}
IGNORE_NAMES = {".env"}


def _ignore_decision(rel: str) -> tuple[bool, str]:
    """Inlined rules. Lift this body in the extract step."""
    parts = rel.split("/")
    for part in parts[:-1]:
        if part.startswith(".") and part not in {".", ".."}:
            return True, "dot"
        if part in IGNORE_DIRS:
            return True, "pattern"
    name = parts[-1]
    if name.startswith(".") and name not in {".", ".."}:
        return True, "dot"
    if name in IGNORE_DIRS or name in IGNORE_NAMES:
        return True, "pattern"
    _, ext = os.path.splitext(name)
    if ext in IGNORE_SUFFIXES:
        return True, "extension"
    return False, "keep"


def messy_walk(root: Path) -> list[dict]:
    """Stand-in for the unsplit scanner."""
    rows: list[dict] = []
    for dirpath, dirnames, filenames in os.walk(root):
        rel_dir = Path(dirpath).relative_to(root).as_posix()
        if rel_dir == ".":
            rel_dir = ""
        dirnames.sort()
        filenames.sort()
        for dirname in list(dirnames):
            rel = f"{rel_dir}/{dirname}" if rel_dir else dirname
            dropped, reason = _ignore_decision(rel)
            rows.append(
                {
                    "path": rel,
                    "kind": "dir",
                    "keep": (not dropped),
                    "reason": reason,
                }
            )
            if dropped:
                dirnames.remove(dirname)
        for filename in filenames:
            rel = f"{rel_dir}/{filename}" if rel_dir else filename
            dropped, reason = _ignore_decision(rel)
            rows.append(
                {
                    "path": rel,
                    "kind": "file",
                    "keep": (not dropped),
                    "reason": reason,
                }
            )
    rows.sort(key=lambda r: (r["path"], r["kind"]))
    return rows


def canonical_bytes(rows: list[dict]) -> bytes:
    text = json.dumps(rows, sort_keys=True, separators=(",", ":"))
    return (text + "\n").encode("utf-8")


def digest(rows: list[dict]) -> str:
    return hashlib.sha256(canonical_bytes(rows)).hexdigest()


def build_fixture(root: Path) -> None:
    files = [
        "README.md",
        "src/app.py",
        "src/app.pyc",
        "src/.hidden.py",
        "src/vendor/lib.py",
        "build/out.js",
        "docs/guide.md",
        "docs/.cache/tmp.md",
        ".git/HEAD",
        "nested/dir/.env",
        "nested/dir/ok.txt",
        "assets/logo.PNG",
        "assets/logo.png",
    ]
    for rel in files:
        path = root / rel
        path.parent.mkdir(parents=True, exist_ok=True)
        path.write_text("x\n", encoding="utf-8")


def main() -> None:
    with tempfile.TemporaryDirectory() as tmp:
        root = Path(tmp)
        build_fixture(root)
        rows = messy_walk(root)
        payload = canonical_bytes(rows)
        hexdigest = digest(rows)
        kept = sum(1 for r in rows if r["keep"])
        dropped = len(rows) - kept
        print(payload.decode("utf-8"), end="")
        print(f"SHA256 {hexdigest}")
        print(f"ROWS {len(rows)}")
        print(f"KEEP {kept} DROP {dropped}")


if __name__ == "__main__":
    main()

Run the script once and capture the printed digest.
Commit that digest as tests/expected_inventory.sha256.
A later pytest compares digest(rows) to that file.

# tests/test_inventory_digest.py
from pathlib import Path

EXPECTED = Path("tests/expected_inventory.sha256").read_text().strip()
FIXTURE = Path("tests/fixtures/inventory_tree")


def test_inventory_digest_matches():
    rows = messy_walk(FIXTURE)
    assert digest(rows) == EXPECTED


def test_vendor_dir_is_dropped():
    rows = {r["path"]: r for r in messy_walk(FIXTURE)}
    assert rows["src/vendor"]["keep"] is False
    assert rows["src/vendor"]["reason"] == "pattern"


def test_readme_is_kept():
    rows = {r["path"]: r for r in messy_walk(FIXTURE)}
    assert rows["README.md"]["keep"] is True
    assert rows["README.md"]["reason"] == "keep"


def test_hidden_file_reason_is_dot():
    rows = {r["path"]: r for r in messy_walk(FIXTURE)}
    assert rows["src/.hidden.py"]["keep"] is False
    assert rows["src/.hidden.py"]["reason"] == "dot"

Digests catch whole-inventory drift in one assert.
Named rows catch inverted reasons on known paths.
Keep both. Neither replaces the other.

Why dirnames mutation belongs in the freeze

os.walk lets the loop prune future descent.
Removing an ignored directory changes later rows.
That prune is observable behavior, not a private trick.

If the extract stops pruning, nested files reappear.
The keep flags on parents may still look correct.
The digest still moves because children return.

Sort dirnames and filenames before recording rows.
Unsorted walks follow inode order on disk.
That order is not a product contract worth hashing.

Smallest extract after the digest is green

Rename _ignore_decision to is_ignored in one patch.
Do not add patterns. Do not drop patterns.
Do not rewrite the os.walk loop in that patch.

def is_ignored(rel: str) -> tuple[bool, str]:
    """Pure predicate. Only extract in this patch."""
    return _ignore_decision(rel)

A true extract inlines the body and deletes the old name.
Call sites inside messy_walk should be the only users.
New callers wait for a second patch with new tests.

If keep flags match and reasons swap, reject the patch.
.env is a dot file and a named ignore.
The frozen reason is dot because that branch runs first.

Decision table for the next patch

Change	Freeze first	Extract now	Wait
Pure is_ignored(rel)	Inventory digest	Yes	—
Glob pattern list	Inventory plus glob tests	No	After ignore
Classifier labels	Inventory plus label map	No	After ignore
Parallel walk	Order-independent digest	No	Separate patch
Real .gitignore parser	This harness is too small	No	New fixture set

Read the table from left to right on every change.
If the freeze column is missing, stop the patch.
The extract column allows one Yes per commit.

Using a free model without trusting it

A model can draft the predicate after the digest exists.
It cannot replace the inventory as the oracle.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free model access and a free server option.
Those two facts are the only product claims used here.
No model names, quotas, or hardware details are asserted.

A practical split looks like the steps below.

1. Keep the harness and fixture on your machine.
2. Send only messy_walk and the digest test.
3. Ask for is_ignored and nothing else.
4. Apply the patch on a throwaway branch.
5. Run the digest test on local bytes.
6. Reject the patch if the hex digest moved.

The free server is useful when the fixture is large.
Copy the harness there. Do not copy secrets.
Bring the digest result back. Do not merge remote trees.

The model output is a diff candidate only.
The hashed inventory is the merge gate.
That order is the whole method.

Limitations

This harness does not implement gitignore semantics.
It skips negation, double-star, and anchored patterns.
Do not cite it as a scanner specification.

os.walk order is directory order plus explicit sorts.
The sample sorts names before it records rows.
Unsorted walks will churn the JSON on every disk.

Windows path case is not frozen in this digest.
logo.PNG and logo.png stay distinct in the fixture.
On a case-folding volume, rebuild the manifest with care.

Symlinks, sockets, and permission errors are omitted.
If the walker follows links, add a link fixture first.
If it skips unreadable dirs, freeze that skip as a reason.

File contents are out of scope for this inventory.
A keep decision is not a parse decision.
Extract parsers only after this digest stays green.

Who should not use this approach

Do not use this when the walker performs network I/O.
Do not use this when ignore rules gate secret detection.
Do not use this as a stand-in for content goldens.

Skip it if the scanner will be replaced entirely.
A new design needs new fixtures, not a frozen mixed loop.
Skip it if the fixture cannot enter version control.

Skip it if the team cannot agree on reason codes.
Unstable reasons poison the digest on every rename.
Keep the enum tiny or drop reason from the hash.

Commands to keep nearby

python characterization_inventory.py | tail -n 3
python characterization_inventory.py > /tmp/inventory.json
pytest tests/test_inventory_digest.py -q
sha256sum tests/expected_inventory.sha256

Compare goldens on the JSON payload, not on log lines.
SHA256, ROWS, KEEP, and DROP are diagnostics.
They do not belong inside the hashed bytes.

Prefer pytest over ad-hoc shells in CI.
Shells hide encoding and trailing newline mistakes.
Pytest makes the digest failure visible in one assert.

Update the expected hash only with a written reason.
Put that reason in the commit body, not the title.
Name every added or removed fixture path in that body.

Closing

Messy walkers fail at the ignore boundary first.
Pin the path inventory. Hash it. Then extract is_ignored.
Leave globbing and classification in the original loop.

If you try this on a mixed scanner, keep the inventory in git.
Treat any model patch as a candidate diff, not a merge.