The Bug Is in the Files: Characterize Filesystem Side Effects Before a Messy Refactor

Conclusion first: in a messy repo, the refactor usually breaks files, not stdout. Teams write golden tests for console output. Almost nobody asserts mode bits, paths, or byte-level file contents. That gap is where the regression hides.

This post is a proposal, not a war story. It gives you a small harness that snapshots a tool's filesystem side effects into a canonical JSON manifest. Then it lets you make one small change and prove the files did not move.

Why stdout goldens leak

stdout is the cheapest contract to test. You capture it, diff it, and move on. Files are harder, because temp paths, timestamps, and modes make every diff noisy.

So teams stop trying. The test asserts that one file exists and has a nonzero size. That test keeps passing while the output layout silently changes underneath it.

For most CLI tools the filesystem is the contract. Exporters, bundlers, migration scripts, and report writers all end at a directory, not a log line.

Step 1 — Pin the base revision, use a throwaway directory

Record the revision before you touch anything. Then run the tool into a directory that nothing else writes to.

git rev-parse HEAD > BASE_REV
rm -rf /tmp/fsrun && mkdir -p /tmp/fsrun
python -m yourtool export --out /tmp/fsrun/out >/tmp/fsrun/stdout.txt 2>&1

Never snapshot a directory your build already dirties. A disposable path makes cleanup a single rm -rf. It also stops .gitignore from hiding files that the tool wrote by accident.

Step 2 — Canonicalize volatile bytes before hashing

Raw hashes churn on every run. Mask the fields that are supposed to vary, and count every mask hit.

#!/usr/bin/env python3
"""fs_contract.py — canonical manifest of a directory's side effects."""
import hashlib, json, os, re, sys
from pathlib import Path

VOLATILE = [
    (re.compile(rb"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?Z?"), b"<TS>"),
    (re.compile(rb"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}"), b"<UUID>"),
    (re.compile(rb"/tmp/[A-Za-z0-9._-]+"), b"<TMP>"),
]
HITS = {}

def canon(data: bytes) -> bytes:
    for rx, repl in VOLATILE:
        data, n = rx.subn(repl, data)
        HITS[rx.pattern.decode()] = HITS.get(rx.pattern.decode(), 0) + n
    return data

def digest(path: Path) -> str:
    return hashlib.sha256(canon(path.read_bytes())).hexdigest()

def manifest(root: Path) -> dict:
    out = {}
    for p in sorted(root.rglob("*")):
        rel, st = p.relative_to(root).as_posix(), p.lstat()
        if p.is_symlink():
            out[rel] = {"kind": "link", "target": os.readlink(p)}
        elif p.is_dir():
            out[rel] = {"kind": "dir", "mode": oct(st.st_mode & 0o777)}
        else:
            out[rel] = {"kind": "file", "mode": oct(st.st_mode & 0o777),
                        "bytes": st.st_size, "sha256": digest(p)}
    return {"files": out, "mask_hits": dict(sorted(HITS.items()))}

if __name__ == "__main__":
    json.dump(manifest(Path(sys.argv[1])), sys.stdout, indent=2, sort_keys=True)

The important part is mask_hits. Masking without counting is silent test deletion. If a timestamp field disappears after your edit, the hash stays identical and the hit count drops to zero.

Diff the hit counts, not only the hashes. A count change means a field vanished or a new one appeared.

Step 3 — Snapshot modes and symlinks too

Content is not the whole contract. A file that flips from 0o644 to 0o755 breaks deployment scripts. A symlink that becomes a real file breaks packaging.

The script above records both. Keep them in the manifest even if your project has never cared before. You will care after the first incident.

Step 4 — Prove the canonicalizer is complete

Run the harness twice on the same revision and diff the two manifests.

python fs_contract.py /tmp/fsrun/out > m1.json
rm -rf /tmp/fsrun && mkdir -p /tmp/fsrun
python -m yourtool export --out /tmp/fsrun/out >/dev/null
python fs_contract.py /tmp/fsrun/out > m2.json
diff m1.json m2.json && echo "canonicalizer complete"

If the diff is non-empty, your canonicalizer is incomplete. Add the missing volatile pattern and repeat. Do not proceed to the refactor until this diff is clean.

Step 5 — One change, then one diff

Now make the smallest behavior-preserving change you can name. Rename a local. Inline a constant. Extract one pure function.

git checkout -b refactor/one-step
# ... edit exactly one thing ...
rm -rf /tmp/fsrun && mkdir -p /tmp/fsrun
python -m yourtool export --out /tmp/fsrun/out >/dev/null
python fs_contract.py /tmp/fsrun/out > after.json
diff -u before.json after.json

An empty diff is the pass condition. A non-empty diff is not automatically a failure. It means you changed a contract, and that change now needs a decision instead of a shrug.

Step 6 — Decide with a table, not with vibes

Change	Allowed with a frozen contract?	Evidence required
Rename a local variable	Yes	Empty diff
Extract a pure function	Yes	Empty diff
Change write order	No	Reviewed diff plus a note on consumers
Rename an output path	No	Consumer search and a migration note
Change a file mode	No	Explicit update to the manifest baseline
Add a new output file	Review	New path documented in the baseline

Paste this table into the pull request. It turns a vague refactor argument into two columns of facts.

Where a free model and a free server actually help

I keep the drafting step narrow. Reading 200 sample outputs by hand to guess which fields are volatile is slow, and it is exactly the kind of pattern-spotting a language model is decent at.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access is useful for proposing candidate volatile patterns, such as timestamps, UUIDs, and temp paths. Treat that output as a draft list only. Every pattern still has to be verified against real bytes with grep -c or the hit counter above, because a wrong regex silently erases a real assertion.

The free server option is useful for one specific job: running the same harness from a clean environment. A fresh machine with an empty /tmp and default locale catches path masking that your laptop hides. Whether that environment fits your repo depends on your dependency footprint, so test it on a small branch first.

Limitations

• Masking hides field disappearance unless you compare hit counts. Always diff mask_hits.
• Byte-level regexes fail on binary artifacts. Zip, sqlite, and images need structural probes instead.
• Ordering inside a file is not captured. Sort JSON keys and CSV rows before hashing, or it is untested.
• Hashing large trees is slow. Add a size cap and skip artifacts over a few megabytes.
• The harness proves stability, not correctness. A file can be consistently wrong.

Who should not use this

If your tool's real contract is a network protocol, snapshot the wire, not the disk. If you have no pinned base revision, pin that first. If the tool writes into your home directory, run it in a container before you build any baseline.

The rule I keep coming back to

Freeze the files. Then move one line. A messy repo becomes tractable when the contract is bytes on disk instead of memory and hope.

Adapt the script above to your own output format, and keep the first change small enough to revert in one command.