Replayable or It Didn't Happen: Five Anti-Patterns in Free-Model Debugging

It failed once. Then it passed. Which run do you trust?

Most people guess. That guess is usually the real bug.

Here is the rule I use now: if you cannot replay a request, you cannot debug it. Replay means same prompt, same parameters, stored response. Everything below follows from that.

I keep hitting the same five anti-patterns when teams debug against free model access and free servers. Each one has a symptom you can spot, a root cause you can name, and a cheap replacement pattern.

Anti-pattern 1: The Single-Run Verdict

Symptom: one green run closes the ticket. One red run opens it again.

Root cause: sampled output is stochastic. A single run measures luck, not behavior.

Replacement: run it k times and store a pass rate. Write the threshold down before you run.

# k=5 is enough to see flakiness. It is not enough for statistics.
for i in 1 2 3 4 5; do python replay.py --run --tag baseline; done
python replay.py --report --tag baseline
# baseline: 3/5 pass (0.60)

A 3/5 today and a 3/5 tomorrow is stable. A 5/5 today and a 2/5 tomorrow is a regression you can prove.

Anti-pattern 2: The Vanishing Request

Symptom: "it worked yesterday" and nobody can rebuild yesterday's prompt.

Root cause: prompts are assembled at runtime from templates, retrieval, and user text. Nothing stores the exact final string.

Replacement: hash model + prompt + params, and append every call to a JSONL log. Replay by hash.

No hash, no bug report. A screenshot is not a request.

Why hash instead of storing the prompt? A hash is safe to commit, share in a ticket, and diff. The prompt stays in the log file, which stays local.

Anti-pattern 3: The Infinite Transcript

The symptom shows up late in a session. Answers get shorter, quality drops, and the quota disappears faster than the token math predicts.

Root cause: every turn re-sends the whole history. Token spend grows with the square of turn count, not linearly.

Replacement: give each turn an explicit budget. Summarize old turns into a fixed block. Trim the retrieved context before you trim the instructions.

# Explicit budget beats implicit growth.
BUDGET = {"system": 400, "retrieved": 1200, "history": 600, "answer": 800}

Track those four numbers per call. If one drifts, you found the quota leak.

Anti-pattern 4: The Assumed-Idle Server

Symptom: the first call times out. The third call is fast. You "fix" a bug that was never yours.

Root cause: free servers go idle. Cold start looks exactly like a model failure from inside your code.

Replacement: measure transport before you judge content. Split errors into timeout, http_status, and bad_content.

# Five probes. Note cold vs warm. Do not judge from one number.
for i in 1 2 3 4 5; do
  curl -sS -o /dev/null -w "probe $i: %{http_code} %{time_total}s\n" "$MC_ENDPOINT"
done

If probe 1 is slow and 2-5 are fast, your timeout is a cold-start problem. Raise the client timeout. Do not touch the prompt.

Anti-pattern 5: The Leaky Replay Log

The symptom is a runs.jsonl in a pull request. The root cause is logging the whole request object because it was convenient.

Replacement: log the hash and a redacted parameter set. Read keys from the environment only. Add the log to .gitignore on day one.

printf 'runs.jsonl\n.env\n' >> .gitignore
git status --short   # should show neither file

The artifact: a replay harness you can actually keep

This is stdlib-only Python. I have not run it against your endpoint; treat it as a template and adjust the request shape to your client.

# replay.py — record once, replay by hash. Stdlib only.
import hashlib, json, os, sys, time, urllib.request

LOG = "runs.jsonl"
ENDPOINT = os.environ["MC_ENDPOINT"]          # base URL from the project page
KEY = os.environ["MC_API_KEY"]                # never hardcode this

def fingerprint(model: str, prompt: str, params: dict) -> str:
    blob = json.dumps({"m": model, "p": prompt, "k": params}, sort_keys=True)
    return hashlib.sha256(blob.encode()).hexdigest()[:16]

def call(model: str, prompt: str, params: dict):
    body = json.dumps({"model": model, "messages": [{"role": "user", "content": prompt}], **params}).encode()
    req = urllib.request.Request(ENDPOINT, data=body, headers={
        "Content-Type": "application/json",
        "Authorization": f"Bearer {KEY}",
    })
    t0 = time.time()
    try:
        with urllib.request.urlopen(req, timeout=90) as r:
            payload = json.loads(r.read())
        return {"ok": True, "payload": payload, "ms": int((time.time() - t0) * 1000)}
    except Exception as e:
        # classify transport failure separately from bad content
        return {"ok": False, "error": type(e).__name__, "ms": int((time.time() - t0) * 1000)}

def record(tag: str, model: str, prompt: str, params: dict):
    fp = fingerprint(model, prompt, params)
    result = call(model, prompt, params)
    row = {"tag": tag, "fp": fp, "model": model, "params": params,
           "prompt_chars": len(prompt), "ok": result["ok"], "ms": result["ms"]}
    with open(LOG, "a", encoding="utf-8") as f:
        f.write(json.dumps(row, ensure_ascii=False) + "\n")
    print(fp, result["ok"], result["ms"], "ms")

def report(tag: str):
    rows = [json.loads(l) for l in open(LOG, encoding="utf-8") if l.strip()]
    rows = [r for r in rows if r["tag"] == tag]
    if not rows:
        return print("no rows for", tag)
    ok = sum(1 for r in rows if r["ok"])
    med = sorted(r["ms"] for r in rows)[len(rows) // 2]
    print(f"{tag}: {ok}/{len(rows)} ok, median {med} ms")

if __name__ == "__main__":
    args = dict(zip(sys.argv[1::2], sys.argv[2::2]))
    if "--run" in sys.argv:
        record(args.get("--tag", "adhoc"), args.get("--model", "default"),
               "Summarize this diff in five bullet points: <paste here>", {"temperature": 0})
    if "--report" in sys.argv:
        report(args.get("--tag", "adhoc"))

Run it twice. The second run gives you a repeatable number instead of a memory.

The decision table I actually use

Evidence	Likely owner	Next move
Same fp, different output, temperature 0	Model or provider	Pin params, re-run 5x, keep the spread
Same fp, ok: false, TimeoutError	Transport / cold start	Preflight, raise timeout, retry once
Different fp on every run	Your code	Log the assembled prompt, fix the builder
Prompt chars jump 3x mid-session	Your code	Add the context budget
4xx on first call only	Auth or quota	Check key scope and remaining quota

That table settles most "is it the model?" arguments in under a minute. Print it. Tape it to the monitor.

Where the free tier fits

The MonkeyCode project is open source, and it currently advertises a free model allotment of 10 million tokens plus a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I use it as the cheap target for the harness above: record a baseline against the free endpoint, keep the fingerprint log, then compare any later change against that baseline. That is a boring use, and boring is the point. Numbers and quotas move, so confirm the current figures on the project page before you size anything around them.

Limitations matter here. A free endpoint has no SLA you can lean on. Model versions can change under the same name. Server behavior on a free host is not production behavior.

Who should not use this approach

• You need sub-second p99 latency. A free server is not that.
• You handle regulated data. A replay log is still a data store.
• You need reproducibility guarantees from the provider. Pin to a paid contract instead.
• You ship on a single sample and call it tested. Fix that habit first.

The checklist

1. Hash every request. Store the hash, not the secret.
2. Run k=5 before you judge anything.
3. Budget context per turn, in writing.
4. Probe the server before you blame the model.
5. Keep runs.jsonl out of git.

So: next time a call fails, ask the boring question first. Can I replay it?

If the answer is no, you have a logging bug, not a model bug. Fix that one first, and the free tier becomes a lab bench instead of a slot machine.

Give the harness a try on one endpoint and one prompt. Ten minutes of recording buys you weeks of not guessing.