48-Hour Field Notes: I Built a Replay Harness Before Trusting a Free Model in a Nightly Job

What do you do when the model that labels your incoming issues costs nothing today and might answer differently tomorrow? I gave myself 48 hours to decide whether a free model tier could sit inside an unattended nightly job, and the first thing I built was not the job at all. It was a harness that records every request and every response, so that a weird classification becomes a diff I can read instead of a mystery I can only retell.

I used MonkeyCode's free model access to drive the probe and its free server option to host the runner, so the nightly job never touched my laptop. Both are operator-supplied availability claims, so confirm the current terms on the product site before you build anything permanent on top of them. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The Job I Actually Wanted To Ship

The goal was boring on purpose: at 02:00 local time, pull new issues, assign one of four labels, and write the label back. No human reads the output before it lands, which is exactly why I did not want to start with the model call. Unattended pipelines fail in the gaps between attempts, and those gaps are invisible unless you keep the raw material.

My working assumption was that a free tier is fine for low-stakes triage and dangerous for anything irreversible. The 48 hours were about testing that assumption instead of arguing about it in a thread, and the harness ended up being more useful than the classifier.

Hour 0–6: Fingerprint Every Request First

You cannot debug a classification you cannot reproduce, so the first artifact was a stable fingerprint over the whole request body. Sort the keys, drop nothing, and hash the canonical JSON so that a prompt edit changes the key.

# probe.py — canonical fingerprint + cassette writer
import hashlib, json, os, time
from pathlib import Path

CASSETTE_DIR = Path(os.environ.get("CASSETTE_DIR", "./cassettes"))
CASSETTE_DIR.mkdir(parents=True, exist_ok=True)

def fingerprint(payload: dict) -> str:
    body = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(body.encode("utf-8")).hexdigest()[:16]

def record(payload: dict, response: dict, tag: str = "run") -> Path:
    path = CASSETTE_DIR / f"{tag}-{fingerprint(payload)}.json"
    path.write_text(json.dumps({
        "payload": payload,
        "response": response,
        "recorded_at": int(time.time()),
    }, indent=2))
    return path

One rule saved me a wasted evening: put a prompt_version field inside the payload, not beside it. If the prompt is not part of the hash, your cassettes will collide across prompt edits and you will chase a phantom regression. Add the model identifier and any sampling parameters to the payload for the same reason.

Hour 6–18: Measure Disagreement, Not Accuracy

Accuracy needs ground-truth labels, and I had none on day one; disagreement needs nothing but repetition. So I ran the same payload five times and asked how often the normalized outputs differed, which is a property of the system rather than a judgment about quality.

# stability.py — disagreement over N identical calls
def normalize(text: str) -> str:
    return " ".join(text.lower().split())

def disagreement_rate(outputs: list[str]) -> float:
    counts: dict[str, int] = {}
    for out in outputs:
        key = normalize(out)
        counts[key] = counts.get(key, 0) + 1
    return 1 - max(counts.values()) / len(outputs)

Treat temperature zero as a strong hint and never as a contract, because some providers ignore seed entirely. My template below is deliberately empty: your prompts and model mix will produce numbers that are not mine, and copying a number you did not measure is how bad benchmarks get born.

Probe	Calls	Disagreement (fill in)	Notes
Raw label only	5		shortest output, least drift
Label + JSON schema	5		drift moves into key order
Label + few-shot examples	5		longer prompt, slower call

Hour 18–30: The Free Server Is a Runner, Not a Database

A free server is a place to execute code, and any file you write there may vanish without warning. So the runner stays stateless, writes cassettes to a scratch directory, and ships them somewhere durable on a schedule; assume the disk is temporary and design the failure path first.

# runner.py — one endpoint, one cassette, no database
import os
from pathlib import Path
from flask import Flask, request, jsonify
from probe import fingerprint, record

app = Flask(__name__)
CASSETTE_DIR = Path(os.environ.get("CASSETTE_DIR", "/tmp/cassettes"))

@app.post("/probe")
def probe():
    payload = request.get_json(force=True)
    payload.setdefault("prompt_version", os.environ.get("PROMPT_VERSION", "dev"))
    response = call_model(payload)   # replace with your provider call
    path = record(payload, response, tag="probe")
    return jsonify({"cassette": path.name, "fingerprint": fingerprint(payload)})

@app.get("/healthz")
def healthz():
    return jsonify({"ok": True, "writable": os.access(CASSETTE_DIR, os.W_OK)})

Point CASSETTE_DIR at the same path in both files, then check the runner from outside before you trust the cron entry. A health endpoint that reports writability is worth more than one that only reports liveness, because a read-only volume looks perfectly healthy right up until the first write.

curl -sS http://localhost:8080/healthz | python -m json.tool
PROMPT_VERSION=2026-11-06-v3 curl -sS -X POST http://localhost:8080/probe \
  -H 'content-type: application/json' \
  -d '{"issue_title":"login fails on Safari","body":"since tuesday"}'

Hour 30–48: A Decision Table Beats a Vibe

After two days of recording, the useful output was not a verdict on free models in general. It was a table that tells me which tasks may run unattended and which need a deterministic rule in code instead of a prompt.

Task	Blast radius if wrong	Repeatable with cassettes?	Verdict
Triage label on a human-reviewed queue	Low	Yes	Free model is reasonable
Auto-closing stale issues	High	Yes	Encode the rule in code, not in a prompt
Draft release notes	Medium	Partly	Model drafts, human edits
Customer-facing status text	Very high	No	Do not start here

What Broke, In Order

1. A prompt edit without a version bump produced cassettes that looked identical but described different behavior.
2. Truncated responses parsed as valid JSON in one attempt and failed in another, so the parser hid the real error.
3. The cron entry fired on server time, not my timezone, and one night of results landed in the wrong bucket.
4. Scratch files disappeared after a restart, which was only obvious because the health check I added reported writability.

Limitations and Who Should Not Use This

If your output reaches customers without review, this workflow is not a substitute for a review step, and no amount of recording will change that. If your task needs strict per-request latency, long-context inputs, or fine-tuned behavior, a free tier is the wrong starting point and the harness will only document your pain more precisely.

Also be honest about the terms you are relying on: availability, quotas, and retention can change, and a nightly job built on today's free allowance may not survive next quarter's policy. Keep the harness provider-agnostic and you can swap the model call without rewriting the pipeline, which is the main reason I put the recorder in front of everything else.

What I Would Repeat

• Build the cassette writer before the classifier, every single time.
• Fingerprint the prompt version inside the payload rather than beside it.
• Measure disagreement before you argue about accuracy with anyone.
• Keep the runner stateless and assume the filesystem is borrowed.

If you want to run this probe without a card on file, MonkeyCode's free model access is where I started, and the harness above will tell you more about your own workload than any review thread will.