Pin Sandbox Path Drift Before One Helper Extract

Pin path side effects before any helper extract. A messy module can still keep green tests. It can still create unexpected new files on disk.

The filesystem snapshot is the real path contract. Change one helper after that contract holds.

Why path drift beats green tests

Most suites pin return values and stdout text. They skip the process working directory. They skip relative writes under that directory. They skip abandoned temp files after exit. Reviewers then miss files that never appear in git.

A helper extract can preserve function output exactly. It can still mkdir a new folder. It can still change a relative report path. It can still leave a .partial file behind. Those failures are path contract failures.

Treat every created path as public behavior. Treat every leftover path as public behavior. Hash contents before you touch names.

Labeled messy module

The module below is a labeled fixture. It is not production telemetry. It writes a report beside the process. It also drops a tempfile in cwd. Both paths are relative. Both paths follow os.getcwd().

# messy_report.py — labeled example, not live production
from __future__ import annotations

import json
import tempfile
from pathlib import Path


def run(payload: dict) -> int:
    scratch = tempfile.NamedTemporaryFile(
        prefix="rpt_",
        suffix=".partial",
        delete=False,
        dir=".",
    )
    scratch.write(b"partial\n")
    scratch.close()

    out_dir = Path("out")
    out_dir.mkdir(exist_ok=True)
    report = out_dir / "report.json"
    report.write_text(
        json.dumps(payload, sort_keys=True),
        encoding="utf-8",
    )
    return 0


if __name__ == "__main__":
    raise SystemExit(run({"ok": True, "n": 2}))

Do not extract write_report yet. The path contract is still unpinned. Return code 0 is not enough evidence.

Artifact: sandbox path snapshot

The artifact is one JSON snapshot file. It records every path under a sandbox root. It records kind, size, mode bits, and sha256. It records leftovers after process exit. Empty directories count as rows.

Place the harness beside the messy module. Keep it read-only toward production trees.

# path_snapshot.py — labeled characterization harness
from __future__ import annotations

import hashlib
import json
import os
import runpy
import stat
import tempfile
from pathlib import Path

IGNORE_NAMES = {".snapshot.json", "__pycache__"}


def _rel(root: Path, path: Path) -> str:
    return path.relative_to(root).as_posix()


def walk_tree(root: Path) -> dict[str, dict]:
    rows: dict[str, dict] = {}
    for dirpath, dirnames, filenames in os.walk(root):
        dirnames[:] = [d for d in dirnames if d not in IGNORE_NAMES]
        base = Path(dirpath)
        for name in sorted(dirnames + filenames):
            if name in IGNORE_NAMES:
                continue
            full = base / name
            rel = _rel(root, full)
            mode = stat.S_IMODE(full.stat().st_mode)
            if full.is_dir():
                rows[rel] = {
                    "kind": "dir",
                    "size": 0,
                    "mode": mode,
                    "sha256": None,
                }
                continue
            data = full.read_bytes()
            rows[rel] = {
                "kind": "file",
                "size": len(data),
                "mode": mode,
                "sha256": hashlib.sha256(data).hexdigest(),
            }
    return rows


def leftover_files(before: dict, after: dict) -> list[str]:
    return sorted(set(after) - set(before))


def snapshot_run(module: Path, argv_payload: str) -> dict:
    with tempfile.TemporaryDirectory(prefix="pathpin_") as tmp:
        root = Path(tmp)
        work = root / "work"
        work.mkdir()
        target = work / module.name
        target.write_bytes(module.read_bytes())
        before = walk_tree(work)
        cwd = os.getcwd()
        try:
            os.chdir(work)
            os.environ["PATH_SNAPSHOT_PAYLOAD"] = argv_payload
            runpy.run_path(str(target), run_name="__main__")
        except SystemExit as exc:
            code = int(exc.code or 0)
        else:
            code = 0
        finally:
            os.chdir(cwd)
        after = walk_tree(work)
        created = leftover_files(before, after)
        return {
            "exit_code": code,
            "created": created,
            "paths": {k: after[k] for k in created},
        }


def main() -> None:
    module = Path("messy_report.py").resolve()
    snap = snapshot_run(module, '{"ok": true, "n": 2}')
    Path(".snapshot.json").write_text(
        json.dumps(snap, indent=2, sort_keys=True),
        encoding="utf-8",
    )
    print(json.dumps({"created": snap["created"]}, indent=2))


if __name__ == "__main__":
    main()

Expected first-run keys stay local to the sandbox. out/report.json must appear. A rpt_*.partial file must appear. No absolute host paths may leak into the JSON.

python path_snapshot.py
python -c "import json; print(json.load(open('.snapshot.json'))['created'])"

Commit .snapshot.json next to the module. That file is the oracle. Later diffs must equal that oracle byte for byte.

Numbered workflow

1. Build an empty sandbox

Copy only the messy module into a temp root. Do not mount the real home directory. Do not reuse /tmp leftovers from prior runs.

python - <<'PY'
from pathlib import Path
import tempfile
p = Path(tempfile.mkdtemp(prefix="pathpin_check_"))
print(p)
PY

2. Capture the before walk

Walk the sandbox before import. Record directories that already exist. Record fixture files you planted. Store that map as before.

3. Run one entrypoint only

Execute runpy.run_path with run_name="__main__". Keep argv and payload frozen. Do not start a second entrypoint in the same sandbox.

4. Capture the after walk

Walk again after process exit. Include files the module forgot to delete. Include empty directories it created. Relative keys only.

5. Promote created rows to tests

Turn created paths into assertions. Assert kind, size, mode, and sha256. Assert leftover names with a stable glob for random tempfile prefixes.

# test_path_contract.py — labeled characterization test
import json
import re
from pathlib import Path

from path_snapshot import snapshot_run

SNAP = json.loads(Path(".snapshot.json").read_text(encoding="utf-8"))
PARTIAL = re.compile(r"^rpt_.+\.partial$")


def test_exit_code_stays_zero():
    got = snapshot_run(Path("messy_report.py"), '{"ok": true, "n": 2}')
    assert got["exit_code"] == SNAP["exit_code"] == 0


def test_report_json_bytes_match():
    got = snapshot_run(Path("messy_report.py"), '{"ok": true, "n": 2}')
    assert got["paths"]["out/report.json"] == SNAP["paths"]["out/report.json"]


def test_partial_tempfile_still_left_behind():
    got = snapshot_run(Path("messy_report.py"), '{"ok": true, "n": 2}')
    leftover = [p for p in got["created"] if PARTIAL.match(p)]
    assert len(leftover) == 1

Random tempfile names will not match as raw strings. Use a prefix and suffix pattern. Pin content hash on the matched file. Do not pin the random middle segment.

6. Extract one helper only

After tests pass twice, extract one write helper. Keep run() as the process owner. Keep tempfile creation in the same function that currently creates it. Do not relocate dir="." in the same patch.

# labeled extract — smallest safe change after the oracle exists
def write_report(payload: dict, out_dir: Path) -> Path:
    out_dir.mkdir(exist_ok=True)
    report = out_dir / "report.json"
    report.write_text(
        json.dumps(payload, sort_keys=True),
        encoding="utf-8",
    )
    return report

Call write_report(payload, Path("out")) from run(). Leave the NamedTemporaryFile block untouched. Re-run the snapshot. Fail the patch if created keys change.

7. Diff the snapshot, not the vibe

Compare JSON with sort_keys=True. Compare sha256, not pretty printed bodies. Compare leftover counts, not log lines.

python path_snapshot.py
python - <<'PY'
import json
from pathlib import Path
new = json.loads(Path(".snapshot.json").read_text())
# keep a copy from git or CI as gold.json
gold = json.loads(Path("gold.snapshot.json").read_text())
assert new["exit_code"] == gold["exit_code"]
assert new["paths"]["out/report.json"] == gold["paths"]["out/report.json"]
print("path contract held")
PY

Any new relative key is a failed extract. Any missing leftover is also a failed extract. Cleanup is a separate change with its own snapshot.

Path contract table

Record decisions in a four-column table. Fill it before the extract. Keep it in the same pull request.

Path pattern	Kind	Must remain	Allowed to change
out/	dir	yes	mode only if documented
out/report.json	file	sha256	none in this patch
rpt_*.partial	file	leftover count = 1	random middle segment
absolute /tmp/...	file	must not appear	n/a

If a cell is unknown, stop. Do not extract. Unknown path cells are missing tests.

Where a free model may edit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free model access and a free server option. Use them after .snapshot.json exists. Paste the snapshot and messy_report.py only. Ask for one helper extract that preserves every path key.

Reject patches that introduce absolute paths. Reject patches that delete the leftover tempfile. Reject patches that rename out/report.json. The free server can run path_snapshot.py as the gate. The model output is not the oracle.

If the snapshot already fails locally, skip the model. Repair the harness first. Models amplify unpinned path drift.

Limitations

This harness does not trace reads outside the sandbox. Network downloads will not appear as rows. Unix sockets and FIFOs are out of scope. Concurrent writers in one sandbox will race the walk.

runpy.run_path is not a full subprocess. It shares the Python interpreter. Import side effects can leak into later tests. Prefer subprocess.run if the module mutates process-global state.

File modes differ across operating systems. Windows will not match Unix 0o644 rows. Keep snapshots OS-specific. Do not commit one gold file for every platform.

Tempfile prefixes can collide. NamedTemporaryFile uses random bytes. Characterization must use a regex. Exact leftover names are the wrong oracle.

Content hashes follow bytes, not parsed JSON. Key order is already pinned by sort_keys=True in the fixture. If the messy module omits sort_keys, pin the raw bytes instead. Do not pretty-print before hashing.

Who should not use this

Do not use this flow for binary installers. Do not use it for tools that must write under $HOME. Do not use it while changing tempfile cleanup and report paths together.

Skip it when the module is already a pure function. Skip it when no relative paths exist. Skip it when your CI cannot create temp directories.

Teams without a freeze step should not batch-extract helpers. Several path moves in one patch hide the first break. One extract per snapshot is the rule.

Close the loop

Keep the gold snapshot in version control. Run the harness on every extract. If created keys drift, revert the helper. Path contracts fail earlier than unit return values. Freeze those contracts before the first rename.