48-Hour Field Notes: The Free Box Restarted, So I Made Every Model Call a File on Disk

I ran a small log-triage batch on a free server over a weekend, and the first thing I learned had nothing to do with models. The box restarted at some point I did not witness, and when I came back the job had happily re-asked every question it had already asked. Does that sound like a bug in your code? It was not — it was a bug in my definition of "done".

The scenario below is the failure mode this harness is built for, and it is written so you can run it against your own logs. The numbers your run prints will be yours, not mine, so treat the hour markers as a narrative device rather than a benchmark. What I want to hand over is the artifact: a cassette layer that turns every model call into a file, plus a replay mode that lets you re-run the whole pipeline with the network switched off.

What the job was supposed to do

The batch read newline-delimited log records and asked a model to map each one onto a small fixed set of incident labels. I wanted one JSON object per record, and I wanted the whole thing to finish overnight without me babysitting it. MonkeyCode's free model access and free server option were the reason I attempted this at all, because I was not going to pay for a weekend experiment on logs nobody was paging me about.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The catch is that "free" changes the shape of your failure budget, not the shape of your problem. A single small box that can restart is a different execution environment from a managed cluster, and the code has to admit that out loud. My first version did not admit anything, which is why hour six was unpleasant.

Hours 0–6: the run that forgot it had run

My first implementation was the obvious loop: read a line, build a prompt, POST it, parse the reply, append to an output file. It worked, it was short, and it was wrong in three separate ways that all showed up at once.

• The output file was appended to after each call, so a restart could duplicate rows rather than resume them.
• A restart re-asked every label question from the top, which is wasteful and also slow on a small box.
• Two asks of the same prompt did not produce identical text, so my "resume" and my "fresh run" disagreed.

That third point is the one people argue about. Setting temperature=0 reduces variance and does not guarantee determinism, so a pipeline that depends on byte-identical answers is a pipeline that will surprise you eventually. I stopped trying to make the model deterministic and started making my own record of what it said durable.

The fix: content-addressed cassettes

The idea is borrowed from HTTP record-and-replay tooling, and it transfers cleanly to chat completions. Hash the whole request payload, use that hash as the filename, and store the response body next to it. On a repeat of the same payload you never touch the network at all.

# cassette.py — content-addressed cache for chat-completion calls
import hashlib, json, os, pathlib, time, urllib.request

ROOT = pathlib.Path(os.environ.get("CASSETTE_DIR", "./cassettes"))
REPLAY = os.environ.get("CASSETTE_REPLAY") == "1"

def key_for(payload: dict) -> str:
    blob = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()
    return hashlib.sha256(blob).hexdigest()[:16]

def call(payload: dict) -> dict:
    ROOT.mkdir(parents=True, exist_ok=True)
    path = ROOT / f"{key_for(payload)}.json"
    if path.exists():
        return json.loads(path.read_text())["response"]
    if REPLAY:
        raise SystemExit(f"cassette miss in replay mode: {path.name}")
    req = urllib.request.Request(
        os.environ["BASE_URL"].rstrip("/") + "/chat/completions",
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json",
                 "Authorization": "Bearer " + os.environ["API_KEY"]},
    )
    with urllib.request.urlopen(req, timeout=90) as r:
        body = json.loads(r.read())
    tmp = path.with_suffix(".tmp")
    tmp.write_text(json.dumps({"payload": payload, "response": body,
                               "at": time.time()}, indent=1))
    os.replace(tmp, path)   # atomic within the same filesystem
    return body

That is the whole trick, and it fits on one screen. Everything else in the field notes is consequences.

Why the key hashes the payload and not the prompt text

Hashing the prompt string alone would collide across model ids, temperatures, system messages, and output-format settings. sort_keys=True gives me a stable serialization, so the same logical request always lands on the same filename. When I change the system message, the key changes, and the old cassettes stay on disk as a record of the previous prompt.

Why the write goes through a temp file

A process can die between open() and the last byte, and a half-written JSON file is worse than no file at all. Writing to .tmp and calling os.replace keeps readers from ever seeing a truncated cassette. If your cassette directory lives on a different filesystem than the temp file, os.replace is not atomic, so keep both in the same tree.

Hours 12–24: replay, and the part that actually saved me

Once calls are files, the pipeline has two modes that should behave identically downstream. Record mode talks to the endpoint and fills the directory; replay mode refuses the network and fails loudly on a miss.

# record
BASE_URL="$MC_BASE_URL" API_KEY="$MC_KEY" CASSETTE_DIR=./cassettes \
  python triage.py --input logs/day1.jsonl --out out/day1.jsonl

# replay, no network, same code path
CASSETTE_DIR=./cassettes CASSETTE_REPLAY=1 \
  python triage.py --input logs/day1.jsonl --out out/day1-replay.jsonl

diff <(sort out/day1.jsonl) <(sort out/day1-replay.jsonl) && echo "identical"

That diff is the test I should have written on day one. If replay and record disagree, my bug is in my own code — parsing, ordering, or output writing — and not in the model. Separating "the model is noisy" from "my pipeline is non-deterministic" is the single most useful thing this layer bought me, and it costs almost nothing to add.

A drift check for the answers themselves

Replay proves my code is stable; it says nothing about whether the model still gives the same label months later. So I keep a small sampler that pulls twenty cassettes, re-asks them live, and counts normalized disagreements. It is not a benchmark, and I would not quote its output as one — it is an early warning that a prompt or a model version moved under me.

# drift.py — sample recorded payloads, re-ask live, report disagreements
def normalize(obj):
    return json.dumps(obj, sort_keys=True).strip()

disagree = 0
for payload in sample_payloads("cassettes", n=20):
    old = normalized_label(payload)
    new = normalize(call_live(payload))
    disagree += (old != new)
print(f"{disagree}/20 labeled differently on re-ask")

Hours 36–48: what the decision table says now

After two days of restarts I stopped treating "free" as a yes-or-no question. Here is the table I would hand to someone starting the same experiment.

Situation	Free model + free box	Paid API	Local model
One-off exploration of unknown logs	good fit	fine	overkill
Nightly batch that can restart and resume	good fit with cassettes	good fit	workable if hardware allows
Request-path latency under a second	no	yes	depends
Log content that cannot leave your network	no	no	yes
You need byte-identical output for audits	no, unless you pin cassettes	no	closer, still not guaranteed

Rows two and five are the ones I keep coming back to. Free infrastructure plus a cassette directory makes a restart cheap; it does not make a model deterministic, and no amount of caching changes that.

What broke anyway, and what I would repeat

Cassettes do not fix everything, and I would rather list the sharp edges than pretend the weekend was clean. Prompt edits silently orphan every old cassette, so a changing prompt and a growing cache directory are a bad pair unless you prune. The directory also grows without bound, and I added a small job that deletes cassettes older than a set age before it becomes somebody's disk-space incident. Cold starts still cost what they cost, and a single box has no failover, so anything resembling an SLA is out of scope.

What I would repeat: putting the network behind a file boundary before scaling up the batch size, and writing the replay diff on the first day instead of the second. What I would skip: the retry wrapper I wrote first, which mostly retried my own malformed requests and taught me nothing.

Who should not use this approach

Skip it if your pipeline is in the request path, because a filesystem lookup and a cold single box are the wrong shape for interactive latency. Skip it if the data cannot legally leave your network, since a remote endpoint is still a remote endpoint. Skip it if you need guaranteed throughput or someone will be paged when the box disappears, because a free box is not a service you can promise to a team. And skip it if you will not maintain the pruning job — an unbounded cache is a slow-motion outage.

If you have run a batch job on a machine that quietly disappears, I would genuinely like to hear what you decided to cache and what you deliberately left uncached. That boundary is the interesting design decision, and mine is still a work in progress.