I Debugged the Wrong Run for 48 Hours. The Folder Had Never Been Empty.

I opened the output directory after a remote job that I believed had finished completely clean. Three files named summary.json sat in that folder, and none of them carried a run identifier. Have you ever grepped through a result that actually belonged to yesterday's unfinished prompt? I had done exactly that, and then I treated the newest mtime as if it were a receipt.

The generation step used a free model, and the execution step used a free server, which sounded simple until the directory started lying. I did not have a cache-key bug this time, and I did not have a missing binary on PATH. I had a quieter problem: leftover files from earlier attempts were sitting in the same place as the run I wanted to trust.

Why a shared dump feels convenient until it is not

I started the way many of us start when a laptop is fine and a remote host is just another shell. I exported OUT=/tmp/out, I let every script write there, and I promised myself I would delete the folder later. Later never arrived during a 48-hour window, because every failed attempt left one more almost-right file behind. Does /tmp/out look harmless in a README? It does, until you cannot say which prompt produced which bytes.

The free server did not isolate my jobs for me, and my laptop copy of the folder was a second source of confusion. I rsynced results back without a run id, then I opened the wrong summary.json in an editor that still had the previous buffer. That is not a model-quality problem. That is a provenance problem I created with a sticky path.

What I tried during the 48 hours

I tried timestamps first, because ls -lt feels like science when you are tired. Newest mtime was a partial file from a job I thought I had killed, and the complete file sat underneath it with an older stamp. I tried renaming outputs with human words like final, final2, and final_really. Those names are not identifiers. They are hope with a .json suffix.

I then routed generation through MonkeyCode because this account already had free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not going to pretend that a free model will stamp a UUID onto every file it proposes. The model will happily emit manifest.json as a filename, and it will happily write relative paths that only make sense inside its own story.

Here is the short list of tactics that failed, in the order I actually tried them:

1. Trusting ls -lt as if mtime were a commit hash for the whole run.
2. Writing into /tmp/out from both the laptop and the free server.
3. Letting the model choose output filenames, including manifest.json.
4. Copying "the latest files" home without copying the directory name.
5. Reading an editor buffer that still held yesterday's summary.json.

None of those failures required a different model. They required a boring rule about directories, and I refused to write that rule until hour thirty.

What actually broke

The first break was a collision I should have predicted on paper. I asked the model to write a small JSON summary, and it also wrote manifest.json, which overwrote my handmade notes about the run. The second break was a relative path that I had treated as stable. Path("output") on my laptop pointed at the repo, and on the free server it pointed at whatever cwd the process manager had chosen that hour.

The third break was a leftover summary.json from an aborted run that my check script treated as current. I had globbed *.json instead of reading a manifest, so the disk told a mixed story and I narrated it as one job. A process can exit, a file can remain, and the next process can read that file as if it were new. If you have ever blamed a model for repeating an old answer, was the model repeating itself, or was your glob repeating the disk?

The rule I should have written on day one

A run does not exist until it has a directory that cannot already exist on disk. That directory holds a run_manifest.json the model is not allowed to name, and every published file must be recorded with a relative path, a byte count, and a sha256. If verification sees an unrecorded file, the run is dirty, and I do not get to discuss the result as if it were clean.

I stopped using manifest.json as the receipt name after the model ate the first one. run_manifest.json is ugly on purpose. Ugly names are less likely to appear in a generated tree, and that small amount of ugliness saved me from a second overwrite.

The isolation harness

The artifact is a small Python tool I now run before any generation step on the laptop or the free server. It creates runs/<uuid>/, writes run_manifest.json, and refuses to record a file outside that tree. I am labeling this as a working local pattern, not as a load-test report, and I am not attaching invented throughput numbers.

#!/usr/bin/env python3
"""isolate_run.py — create a unique workdir and refuse to start without it."""
from __future__ import annotations

import argparse
import hashlib
import json
import os
import sys
import time
import uuid
from pathlib import Path
from typing import Any


def utc_now() -> str:
    return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())


def sha256_file(path: Path) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        for chunk in iter(lambda: handle.read(65536), b""):
            digest.update(chunk)
    return digest.hexdigest()


def create_run(root: Path) -> dict[str, Any]:
    run_id = str(uuid.uuid4())
    work = root / "runs" / run_id
    work.mkdir(parents=True, exist_ok=False)
    manifest = {
        "run_id": run_id,
        "started_at": utc_now(),
        "pid": os.getpid(),
        "host": os.environ.get("HOSTNAME")
        or os.environ.get("COMPUTERNAME")
        or "unknown",
        "cwd": str(work.resolve()),
        "files": [],
    }
    path = work / "run_manifest.json"
    path.write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8")
    return manifest


def record_file(manifest_path: Path, written: Path) -> None:
    manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
    written = written.resolve()
    work = Path(manifest["cwd"])
    if work not in written.parents and written != work:
        raise SystemExit(f"refusing to record file outside the run: {written}")
    entry = {
        "path": str(written.relative_to(work)),
        "sha256": sha256_file(written),
        "bytes": written.stat().st_size,
        "recorded_at": utc_now(),
    }
    manifest["files"].append(entry)
    manifest_path.write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8")


def verify_run(manifest_path: Path) -> int:
    manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
    work = Path(manifest["cwd"])
    errors = 0
    recorded = {item["path"] for item in manifest["files"]}
    on_disk: list[str] = []
    for path in work.rglob("*"):
        if not path.is_file():
            continue
        rel = str(path.relative_to(work))
        if rel == "run_manifest.json":
            continue
        on_disk.append(rel)
        if rel not in recorded:
            print(f"UNRECORDED {rel}", file=sys.stderr)
            errors += 1
    for item in manifest["files"]:
        path = work / item["path"]
        if not path.is_file():
            print(f"MISSING {item['path']}", file=sys.stderr)
            errors += 1
            continue
        if sha256_file(path) != item["sha256"]:
            print(f"HASH_MISMATCH {item['path']}", file=sys.stderr)
            errors += 1
    return errors


def main() -> None:
    parser = argparse.ArgumentParser()
    sub = parser.add_subparsers(dest="cmd", required=True)
    start = sub.add_parser("start")
    start.add_argument("--root", type=Path, required=True)
    rec = sub.add_parser("record")
    rec.add_argument("--manifest", type=Path, required=True)
    rec.add_argument("--file", type=Path, required=True)
    ver = sub.add_parser("verify")
    ver.add_argument("--manifest", type=Path, required=True)
    args = parser.parse_args()
    if args.cmd == "start":
        print(create_run(args.root)["cwd"])
        return
    if args.cmd == "record":
        record_file(args.manifest, args.file)
        return
    raise SystemExit(verify_run(args.manifest))


if __name__ == "__main__":
    main()

Starting a run looks like this on either host. Notice that I never cd into /tmp/out anymore, and I never let the shell pick a sticky dump directory.

export RUN_ROOT="$HOME/work"
WORKDIR="$(python3 isolate_run.py start --root "$RUN_ROOT")"
export RUN_WORKDIR="$WORKDIR"
export RUN_MANIFEST="$WORKDIR/run_manifest.json"
cd "$RUN_WORKDIR"
echo "host=$(hostname) cwd=$PWD run_manifest=$RUN_MANIFEST"

The generation script then refuses to start without that environment. This is the whole point of the 48 hours: make the missing directory a hard error, not a surprise at review time.

import os
from pathlib import Path

def require_run_env() -> Path:
    raw = os.environ.get("RUN_WORKDIR")
    if not raw:
        raise SystemExit("RUN_WORKDIR is missing; refusing to write anywhere")
    work = Path(raw).resolve()
    marker = work / "run_manifest.json"
    if not marker.is_file():
        raise SystemExit(f"no run_manifest.json in {work}")
    return work

If that guard had existed on hour one, I would not have spent hour thirty reading a file that belonged to a different host. Would you rather fail closed before the model runs, or explain three summaries after the fact?

A verification pass that does not need a dashboard

After the job, I do not open files at random and I do not trust the editor tabs. I run verify and I read the stderr lines as a checklist, not as decoration. UNRECORDED means something wrote around my helper. MISSING means the manifest is bragging. HASH_MISMATCH means the bytes changed after I recorded them, which is either a second writer or a pretty-printer that rewrote spacing.

python3 isolate_run.py record --manifest "$RUN_MANIFEST" --file "$RUN_WORKDIR/summary.json"
python3 isolate_run.py verify --manifest "$RUN_MANIFEST"
echo "verify_exit=$?"

I also keep a tiny decision table next to the repo, because three-in-the-morning me will not remember these distinctions without a table.

Symptom	First check	Do not do
Several summary.json files in one folder	You used a shared dump	Do not pick the newest mtime
Verify says UNRECORDED	A tool wrote around record_file	Do not hand-edit the manifest
Verify says MISSING	The job logged a path it never wrote	Do not rsync a glob home
Verify says HASH_MISMATCH	A second process rewrote the file	Do not "fix" JSON by pretty-printing
Host in the manifest is not the host you are on	You copied files without the run directory	Do not mix laptop cwd and server cwd
manifest.json overwrote your notes	The model chose a receipt name	Do not let the model name receipts

A twenty-minute pass I will actually repeat looks like this:

1. Start a UUID directory with isolate_run.py start and export RUN_WORKDIR.
2. cd there, print host and cwd, and refuse to continue if either value looks wrong.
3. Generate only into that directory, with the model blocked from naming run_manifest.json.
4. Record each file the job intended to publish, then run verify.
5. Copy the entire runs/<uuid> folder home; never copy a glob of *.json.

Labeled example, not a production benchmark: I keep a local unit test that only checks the refusal paths. It does not prove a free model is stable. It proves the harness will not record a file that escaped the tree.

import json
import tempfile
import unittest
from pathlib import Path

import isolate_run

class IsolateRunTests(unittest.TestCase):
    def test_start_creates_manifest(self) -> None:
        with tempfile.TemporaryDirectory() as raw:
            root = Path(raw)
            manifest = isolate_run.create_run(root)
            work = Path(manifest["cwd"])
            self.assertTrue((work / "run_manifest.json").is_file())
            self.assertEqual(manifest["files"], [])

    def test_record_rejects_outside_tree(self) -> None:
        with tempfile.TemporaryDirectory() as raw:
            root = Path(raw)
            manifest = isolate_run.create_run(root)
            marker = Path(manifest["cwd"]) / "run_manifest.json"
            outsider = root / "nope.json"
            outsider.write_text("{}"\n", encoding="utf-8")
            with self.assertRaises(SystemExit):
                isolate_run.record_file(marker, outsider)

if __name__ == "__main__":
    unittest.main()

Wait, that outsider fixture is easier to read if the string is ordinary JSON. The test still makes the same point: the helper must fail closed when a path leaves the run.

What I would repeat

I would repeat the UUID directory, the ugly manifest name, and the outside-the-tree refusal on both hosts. I would repeat copying the entire runs/<uuid> folder home, never a glob of *.json that looks complete in a hurry. I would repeat printing host and cwd at the top of every log line, because mixed hosts are how laptop-truth and server-truth quietly merge.

I would not repeat final.json. I would not repeat a shared dump that I planned to clean later. I would not ask the model to invent the output layout, because it will invent a layout that collides with the last run. If you already have free model access and a free server option, run the isolator on that host before the next batch, not after you have three summaries.

Who should skip this

Skip this harness if you already launch each job in a fresh container with a fresh volume and you already archive that volume. Skip it for a one-line REPL experiment you will throw away in ten minutes and never quote later. Skip it if you cannot write a work directory on the host you are using, because the whole point is a directory that starts empty and stays named.

This pattern will not make a free model deterministic, and it will not turn a vague prompt into a careful patch. It will not replace code review, and it will not tell you whether the generated code is any good. It only answers a smaller question that kept wasting my 48 hours: which files belong to the run I am talking about right now?

If your output folder can survive a second run without a UUID, do you actually know which files you are reading?