Snapshot File Deltas Before You Extract Glue

Do not extract glue until side effects are pinned. Messy modules leak behavior through files, env, and cwd. A function signature is not a characterization test.

Why naive extracts fail

AI diffs look clean when tests are missing. Glue code still writes caches, logs, and temp files. Reviewers then ship a rename that changes process behavior.

Cheap code generation does not reduce the characterization cost. Technical debt remains in implicit state, not syntax. You need a ledger before the first extract.

Untested glue often reads hidden config from the process. It may chdir, mkdir, or rewrite a lockfile. Those actions never appear in the public return value.

Return values also miss mkdir side effects on empty input. Missing directories can change later commands in the same job. That is still user-visible behavior, not an internal detail.

What a side-effect ledger is

A ledger is a frozen record of process behavior. It is not a unit test of internal helpers. It pins observable results for a fixed fixture pack.

Record five observable channels on every fixture run. Capture exit code, stdout, stderr, file deltas, and env keys. Ignore wall-clock time unless the contract requires it.

Hash file contents, not only the file names. Store relative paths instead of absolute sandbox paths. Drop volatile fields like pid, dates, and hostnames.

Normalize stdout by stripping any sandbox path prefix. Keep the newline style identical across all runs. Sort JSON object keys before hashing cache files.

The messy target

Consider a report glue script with no tests. It reads CSV from stdin and writes a cache file. It also prints a summary line and an exit code.

The listing below is a labeled example, not production code. It mixes parsing, caching, and printing in one function. That mix is the core extract hazard.

# proposal: messy glue, not a live codebase
import csv, hashlib, json, os, sys

CACHE = os.environ.get("REPORT_CACHE", ".cache/report.json")

def run():
    os.makedirs(os.path.dirname(CACHE) or ".", exist_ok=True)
    rows = list(csv.DictReader(sys.stdin))
    total = sum(int(r.get("amount") or 0) for r in rows)
    payload = {"count": len(rows), "total": total}
    digest = hashlib.sha256(
        json.dumps(payload, sort_keys=True).encode()
    ).hexdigest()[:12]
    with open(CACHE, "w", encoding="utf-8") as f:
        json.dump({**payload, "digest": digest}, f)
    print(f"rows={payload['count']} total={payload['total']} digest={digest}")
    return 0

if __name__ == "__main__":
    raise SystemExit(run())

Do not split the run function just yet. First pin what the whole process does. Then change one seam only after that pin.

Artifact: a copyable ledger harness

The harness below is a proposal you can copy. It runs one command inside a temp workspace. It writes a stable JSON ledger for later diffing.

# proposal: characterization harness, unexecuted here
import hashlib, json, os, shutil, subprocess, tempfile
from pathlib import Path

VOLATILE_ENV = {"PWD", "OLDPWD", "SHLVL", "SSH_AUTH_SOCK", "TERM"}

def hash_tree(root: Path) -> dict:
    out = {}
    for path in sorted(root.rglob("*")):
        if path.is_file():
            rel = str(path.relative_to(root)).replace("\\", "/")
            out[rel] = hashlib.sha256(path.read_bytes()).hexdigest()
    return out

def capture(cmd, stdin_bytes, extra_env):
    sandbox = Path(tempfile.mkdtemp(prefix="ledger-"))
    try:
        env = {
            key: value
            for key, value in os.environ.items()
            if key not in VOLATILE_ENV
        }
        env.update(extra_env)
        proc = subprocess.run(
            cmd,
            input=stdin_bytes,
            cwd=sandbox,
            env=env,
            capture_output=True,
        )
        return {
            "exit": proc.returncode,
            "stdout": proc.stdout.decode("utf-8", "replace"),
            "stderr": proc.stderr.decode("utf-8", "replace"),
            "files": hash_tree(sandbox),
            "env_subset": {key: env.get(key) for key in sorted(extra_env)},
        }
    finally:
        shutil.rmtree(sandbox, ignore_errors=True)

def write_ledger(path, cases):
    Path(path).write_text(json.dumps(cases, indent=2, sort_keys=True) + "\n")

Add a tiny fixture runner next to the harness. Keep the fixtures boring, small, and fully deterministic. Three fixture cases beat one lucky happy path.

# proposal: fixture pack for report.py
import sys
from pathlib import Path

REPORT = str(Path("report.py").resolve())
CASES = [
    {
        "name": "empty_stdin",
        "stdin": b"",
        "env": {"REPORT_CACHE": "out/report.json"},
    },
    {
        "name": "two_rows",
        "stdin": b"amount\n10\n5\n",
        "env": {"REPORT_CACHE": "out/report.json"},
    },
    {
        "name": "missing_amount",
        "stdin": b"name\nalice\n",
        "env": {"REPORT_CACHE": "out/report.json"},
    },
]

def main(out_path):
    cmd = ["python", REPORT]
    ledgers = {}
    for case in CASES:
        ledgers[case["name"]] = capture(cmd, case["stdin"], case["env"])
    write_ledger(out_path, ledgers)

if __name__ == "__main__":
    main(sys.argv[1] if len(sys.argv) > 1 else "ledger.before.json")

Store the before ledger inside the git history. Treat that committed file as a characterization oracle. Diff the ledger file after every candidate extract.

Numbered workflow

Follow these seven steps in strict order. Skip none of these steps on messy glue.

1. Inventory the process contract

Inventory the process contract in one short note. List stdin, env keys, output files, and exit codes. Do not list internal helper names just yet.

Write the note as a bullet list in the PR. Name each env key and its default value. Name each file path the process may create.

2. Build the fixture pack

Build a fixture pack from real sample inputs. Prefer anonymized production-like rows over synthetic ones. Keep each fixture file under a few kilobytes.

Include empty stdin, two valid rows, and a missing column. Do not include live hostnames or access tokens. Redact any customer field before the file lands.

3. Capture the before ledger

Run the harness and commit the before ledger. Confirm the empty, happy, and malformed cases exist. Confirm that file hashes are relative and stable.

Commit harness code with the ledger in one snapshot. That pair is the oracle for later diffs. Do not fold extra formatting into this snapshot.

4. Freeze the repo

Freeze the git repo at that ledger commit. Do not format, rename, or extract in the same commit. The ledger commit must contain zero behavior edits.

Tag the commit if your team likes movable pointers. A local branch name is enough for a solo run. The point is a restore target after a bad extract.

5. Propose one extract

Propose the smallest extract that preserves the ledger. One function move is enough for this step. Do not relocate caches or change env names.

Move only the cache write into a helper. Leave CSV parsing inside the original run function. Leave stdout formatting inside the original run function.

# proposal: smallest extract after the ledger pin
import csv, hashlib, json, os, sys

CACHE = os.environ.get("REPORT_CACHE", ".cache/report.json")

def write_cache(path, record):
    os.makedirs(os.path.dirname(path) or ".", exist_ok=True)
    with open(path, "w", encoding="utf-8") as handle:
        json.dump(record, handle)

def run():
    rows = list(csv.DictReader(sys.stdin))
    payload = {
        "count": len(rows),
        "total": sum(int(r.get("amount") or 0) for r in rows),
    }
    digest = hashlib.sha256(
        json.dumps(payload, sort_keys=True).encode()
    ).hexdigest()[:12]
    write_cache(CACHE, {**payload, "digest": digest})
    print(f"rows={payload['count']} total={payload['total']} digest={digest}")
    return 0

6. Capture the after ledger

Re-run the same harness into the after ledger. Diff the two JSON files with a structured tool. Any extra JSON key is a failed characterization.

python harness.py ledger.before.json
git add ledger.before.json report.py harness.py
git commit -m "test: pin report glue side-effect ledger"

# smallest extract only, after the pin commit
python harness.py ledger.after.json
python - <<'PY'
import json
from pathlib import Path
before = json.loads(Path("ledger.before.json").read_text())
after = json.loads(Path("ledger.after.json").read_text())
print("match" if before == after else "mismatch")
if before != after:
    for name in sorted(set(before) | set(after)):
        if before.get(name) != after.get(name):
            print("case", name)
PY

7. Keep or revert

Keep the extract only when the ledgers match. Revert the whole extract on the first mismatch. Do not "fix forward" inside the same change.

A later commit may fix a real product bug. That fix needs new fixtures and a new pin. Do not hide a bugfix inside a rename.

Decision table

Use this table before you accept a diff.

Signal	Keep the extract	Revert the extract
Exit codes	Identical per fixture	Any fixture changed
stdout / stderr	Byte-stable after path rewrite	New warnings or missing lines
File set	Same relative paths	Extra cache or dropped file
File hashes	Same digest per path	Any digest changed
Env contract	Same keys read and written	New required key
Diff size	One symbol moved	Helpers, names, and cache all move

If three or more rows say revert, stop. The change is not the smallest safe extract. Split the work and then pin again.

Byte-stable stdout still fails when a new file appears. New files are process behavior, not formatting noise. Treat those new files as a hard mismatch.

Using a model after the pin

A coding model can suggest the extract later. The model still cannot invent the characterization oracle. Generate patch candidates only against a committed ledger.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option.

Use the server to run the harness on a clean workspace. Use a free model to propose one extract after the pin.

Do not ask the model to invent fixtures from memory. Paste the inventory note and the ledger schema instead. Reject any patch that touches cache paths and helpers together.

Limitations

This ledger ignores in-process mocks and thread timing. It will miss behavior that never touches stdout or files. Shared network calls need a different freeze, such as recorded HTTP.

Hashing whole trees can hide permission-only file changes. Empty directories may disappear from the file map. Binary files need an explicit allow list on large repos.

Fixture packs go stale when product rules change. A matching ledger can still encode a bug. Characterization preserves current behavior, including known bad behavior.

The harness also assumes a single process per fixture. Child daemons will outlive the sandbox cleanup step. Stop those child jobs before hashing the tree.

Who should not use this

Do not use this on greenfield modules with real tests. Do not use this when the contract is a public API suite. Do not use this for security-sensitive parsers without a fuzzer.

Skip it when the glue is a thin wrapper over a stable library. Skip it when you cannot sandbox file writes. Skip it when fixtures would include live secrets.

Skip it when the script must talk to a live billing API. Recorded side effects would then include third-party state. That class of glue needs contract tests, not file hashes.

Close

Pin the side-effect ledger before any extract. Extract one seam only after the ledgers exist. Then diff the two JSON ledger files.

That strict order is the entire safety method. The coding model remains optional in this workflow. The characterization oracle is not optional here.

If you run this on a free workspace, keep the ledger in git. Leave the model out until the oracle exists.