Pin a Working-Tree Oracle Before You Edit a Messy Package

A messy package fails refactors at the filesystem, not the return value. Pin one command against a fixture workspace before any edit. Accept a patch only when that working-tree oracle stays identical.

Why unit tests miss messy packages

Unit tests pin return values and miss the rest of the command. Messy packages write caches, logs, and sibling files without ceremony. They also read cwd, leftover env vars, and hidden config files.

A green helper test does not prove the package is safe. The real contract is the tree after one user command. That tree is the oracle this workflow records and checks.

Cheap model diffs make this gap more expensive, not less. Cheap edits still move files, log lines, and cache layouts. Characterization belongs on the command, not on the helper.

What this workflow freezes

Freeze three things only: command, fixture tree, and side channels. Side channels here means stdout, stderr, return code, and file hashes. Leave private helpers unfrozen so a later rewrite can move them.

Do not start inside the densest file in the package. Start at the command a human actually runs during work. The smallest safe change sits behind that frozen command boundary.

Git stores both the fixture directory and the oracle JSON. The oracle file is data, not a substitute for review. Reviewers should read the hashed paths before they read the patch.

Artifact: a working-tree oracle harness

The script below records one command against one fixture directory. It stores argv, return code, stdout, stderr, and a sorted file manifest. A later check mode compares a fresh run to that JSON snapshot.

Treat the script as a local tool, not a published benchmark. Run it on your machine before you invite any coding model.

#!/usr/bin/env python3
"""Record or check one command against a fixture working tree."""
from __future__ import annotations

import hashlib
import json
import subprocess
import sys
from pathlib import Path

ORACLE_NAME = "working_tree_oracle.json"


def hash_file(path: Path) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        for chunk in iter(lambda: handle.read(65536), b""):
            digest.update(chunk)
    return digest.hexdigest()


def manifest(root: Path) -> list[dict]:
    rows: list[dict] = []
    for path in sorted(root.rglob("*")):
        if not path.is_file() or path.name == ORACLE_NAME:
            continue
        rel = path.relative_to(root).as_posix()
        rows.append(
            {
                "path": rel,
                "sha256": hash_file(path),
                "size": path.stat().st_size,
            }
        )
    return rows


def run_command(root: Path, argv: list[str]) -> dict:
    proc = subprocess.run(argv, cwd=root, capture_output=True, text=True)
    return {
        "argv": argv,
        "returncode": proc.returncode,
        "stdout": proc.stdout,
        "stderr": proc.stderr,
        "files": manifest(root),
    }


def split_argv(argv: list[str]) -> tuple[str, Path, list[str]]:
    if len(argv) < 3 or "--" not in argv:
        raise SystemExit("usage: oracle.py record|check FIXTURE -- cmd...")
    mode, fixture = argv[0], Path(argv[1]).resolve()
    command = argv[argv.index("--") + 1 :]
    if mode not in {"record", "check"} or not command:
        raise SystemExit("usage: oracle.py record|check FIXTURE -- cmd...")
    return mode, fixture, command


def print_diff(expected: dict, actual: dict) -> None:
    for key in ("argv", "returncode", "stdout", "stderr"):
        if expected.get(key) != actual.get(key):
            print(f"MISMATCH {key}")
            print("expected:", repr(expected.get(key)))
            print("actual:  ", repr(actual.get(key)))
    exp_files = {row["path"]: row for row in expected.get("files", [])}
    act_files = {row["path"]: row for row in actual.get("files", [])}
    for path in sorted(set(exp_files) | set(act_files)):
        if exp_files.get(path) != act_files.get(path):
            print(f"MISMATCH file {path}")
            print("expected:", exp_files.get(path))
            print("actual:  ", act_files.get(path))


def main() -> None:
    mode, fixture, command = split_argv(sys.argv[1:])
    if not fixture.is_dir():
        raise SystemExit(f"missing fixture directory: {fixture}")
    snap = run_command(fixture, command)
    oracle_path = fixture / ORACLE_NAME
    if mode == "record":
        oracle_path.write_text(json.dumps(snap, indent=2, sort_keys=True) + "\n")
        print(f"recorded {oracle_path}")
        return
    if not oracle_path.is_file():
        raise SystemExit(f"missing oracle file: {oracle_path}")
    expected = json.loads(oracle_path.read_text())
    if snap == expected:
        print("oracle match")
        return
    print_diff(expected, snap)
    raise SystemExit(1)


if __name__ == "__main__":
    main()

Treat absolute timestamps inside file bodies as a known limitation. Normalize those fields in the command under test when possible. The harness hashes bytes, so unstable bytes will fail the check.

A second module below builds a tiny messy package for practice. It writes a cache file, prints a banner, and honors cwd config. Use it to learn the workflow before you touch a real tree.

# messy_pkg/__main__.py
from pathlib import Path
import json
import sys


def main(argv: list[str]) -> int:
    cwd = Path.cwd()
    config = json.loads((cwd / "messy.json").read_text())
    cache = cwd / ".messy_cache"
    cache.mkdir(exist_ok=True)
    payload = {
        "ok": True,
        "name": config.get("name", "anon"),
        "n": len(argv),
    }
    (cache / "last.json").write_text(json.dumps(payload, sort_keys=True) + "\n")
    print(f"built {payload['name']}")
    print("cache-ready", file=sys.stderr)
    return 0


if __name__ == "__main__":
    raise SystemExit(main(sys.argv[1:]))

{
  "name": "fixture-demo"
}

mkdir -p fixture
cp messy.json fixture/
PYTHONPATH=. python oracle.py record ./fixture -- python -m messy_pkg build
# restore fixture, then:
PYTHONPATH=. python oracle.py check ./fixture -- python -m messy_pkg build

1. Pick one command

Choose one command that exercises the messy package end to end. Prefer a CLI entry or a small __main__ module over a helper. Skip networked commands until you can stub the network boundary.

Write the exact argv in the oracle file, not in docs. Argv drift is a contract change even when Python files look fine. Pin the interpreter the same way you pin the fixture data.

2. Build a fixture workspace

Copy a reduced data set into a disposable fixture directory. Keep that directory small enough to read in one sitting. Commit the fixture with the oracle file after the first record.

Strip secrets from the fixture before the first record run. The oracle JSON will capture stdout, so secrets would leak there. Use fake hostnames and fake keys inside the reduced data set.

3. Record the oracle

Run the harness in record mode from a clean fixture copy. Inspect stdout, stderr, and the hashed file list by hand. Delete accidental junk files before you accept the snapshot.

Re-run record mode twice and confirm the JSON bytes match. A mismatch here means the command is not yet characterizable. Fix nondeterminism before you discuss any later refactor candidate.

4. Rank internal files

List files the command imports, then rank them by blast radius. Blast radius means how many other files import the candidate. The smallest safe change is the lowest-radius file that still compiles.

Do not rank by line count or by how ugly the file looks. Ugly files with many importers are extraction work, not first edits. Ugly files with one importer are the usual first patch target.

A cheap import census in Python can use this short grep.

python - <<'PY'
from pathlib import Path
import collections
import re

root = Path("messy_pkg")
pattern = re.compile(r"^(?:from|import)\s+([a-zA-Z0-9_\.]+)", re.M)
importers = collections.defaultdict(set)
for path in root.rglob("*.py"):
    text = path.read_text(encoding="utf-8", errors="replace")
    for match in pattern.finditer(text):
        importers[match.group(1)].add(path.as_posix())
for mod, files in sorted(importers.items(), key=lambda kv: (-len(kv[1]), kv[0])):
    print(f"{len(files):3} {mod}")
PY

5. Change one internal file

Edit one file and do not rename packages in the same patch. Do not retouch fixture data while you rewrite that file. Keep import paths stable so the command still finds the module.

If the model also rewrites tests, drop that part of the diff. The oracle already owns the behavior, so extra tests can wait. One file means one file, including generated code and lockfiles.

6. Check the oracle

Restore the fixture, then run the harness in check mode. A mismatch means the command contract moved, so revert the file. A match means the internal rewrite stayed behind the frozen boundary.

Keep the failing JSON pair when a check rejects a patch. Diff stdout first, then stderr, then the file manifest paths. Most false refactors show up as one extra cache file.

Decision table for the smallest safe change

Use the table below before you accept a candidate patch.

Candidate change	Import paths move	Fixture bytes move	Argv moves	Action
Rewrite one helper in place	no	no	no	run check
Extract a sibling module	yes	no	no	defer
Rename the package	yes	no	often	reject
Reformat fixture JSON	no	yes	no	reject
Add a CLI flag	no	no	yes	reject
Silence a log line	no	no	no	treat as contract

Read the last row twice before you clean up logs. Stderr text is part of the oracle in this workflow. If operators parse those lines, the log line is public API.

After the oracle is green

A coding model can draft the one-file patch after that gate. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option for that draft.

Keep the oracle local and the prompt narrow to one file path. Paste the failing census and the oracle paths, not the whole repo. Reject any patch that touches the fixture directory or the argv.

If you try the harness, keep the oracle in git before any model run.

Limitations

This method does not prove correctness against a written spec. It only proves the command still produces the same side channels. That is characterization, which can freeze bugs you already ship.

Hashes fail when files embed clocks, PIDs, or absolute paths. Sort order in the manifest is stable; file internals may not be. Normalize those fields inside the command before you record.

Parallel writes can still shuffle non-file side channels like logs. If stderr interleaves, the oracle will flake under load. Serialize the command or drop this method for that package.

Binary assets bloat the fixture and slow every check run. Prefer text fixtures unless the command must read a real blob. Large blobs also hide the paths you should read by hand.

Who should not use this

Do not use this on live production commands that send mail. Do not use this on commands that charge a cloud account. Do not use this when you cannot restore the fixture each run.

Skip it if the package has no single command-shaped entry. Library-only packages need a tiny driver script first of all. Without that driver, you are back to function-level guesses.

Skip it if reviewers will not read the hashed path list. An unread oracle is theater and will rubber-stamp bad edits. The workflow needs a human on the mismatch, not a dashboard.

Close

Commit the oracle file before any internal rewrite work starts. Then allow exactly one internal path to change per patch. If the check fails, the command contract moved, so stop.