48-Hour Field Notes: I Gated a Free-Model Batch Job on Agreement Rate, Not on Schema

I spent 48 hours pointing a free model endpoint at 6,400 lines of production logs, and every single response was valid JSON.

That was the problem. My validator checked shape, my pipeline checked exit codes, and both of them told me the batch had succeeded. Then I sampled forty rows by hand and found that roughly one in five labels had quietly drifted to a neighbouring category, which meant my downstream alerting had been trained on noise for two days.

So this is a field note about the check I should have written first: a golden-set agreement gate that runs before the batch, not after it.

What I actually tried

I had a boring, unglamorous task: classify log lines into a fixed label set so I could route them to the right runbook. The rules-based version had grown to 300 lines of regex and was wrong in ways I could no longer read, so a model felt like the cheaper maintenance path.

I did not want to spend money on an experiment I might throw away, which is exactly the case where MonkeyCode's free model access and free server option fit my constraint. I ran the classifier on the free server, kept the golden set on my laptop, and treated the whole thing as a two-day spike rather than infrastructure.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Before you copy anything below, check the project's own docs for current limits on token allowance and server specs, because I am not going to quote numbers I cannot keep accurate for you.

The artifact: a gate that runs before the batch

The harness is about 120 lines of stdlib Python. It reads a golden.jsonl file of hand-labelled lines, asks the endpoint for a label, and refuses to let the batch start if agreement falls under a floor you choose.

# gate.py — exit 1 means do not run the batch
import json, os, sys, urllib.request

LABELS = ["auth", "rate_limit", "timeout", "schema", "other"]
MIN_AGREE = 0.85

def classify(text):
    body = json.dumps({
        "model": os.environ["MODEL_ID"],
        "temperature": 0,
        "messages": [
            {"role": "system",
             "content": "Label the log line. Reply with exactly one of: " + ", ".join(LABELS)},
            {"role": "user", "content": text},
        ],
    }).encode()
    req = urllib.request.Request(
        os.environ["BASE_URL"].rstrip("/") + "/chat/completions",
        data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": "Bearer " + os.environ["API_KEY"]},
    )
    with urllib.request.urlopen(req, timeout=60) as r:
        out = json.load(r)["choices"][0]["message"]["content"].strip()
    return out if out in LABELS else "other"

def gate():
    rows = [json.loads(line) for line in open("golden.jsonl") if line.strip()]
    hits, per = 0, {}
    for row in rows:
        got = classify(row["text"])
        ok = got == row["label"]
        hits += ok
        bucket = per.setdefault(row["label"], [0, 0])
        bucket[0] += ok
        bucket[1] += 1
    for label, (h, n) in sorted(per.items()):
        print(f"{label:12s} {h}/{n}  recall={h / n:.2f}")
    agree = hits / len(rows)
    print(f"overall agreement {agree:.2f} (floor {MIN_AGREE})")
    return agree >= MIN_AGREE

if __name__ == "__main__":
    sys.exit(0 if gate() else 1)

Run it as a cron job or a pre-flight step, and wire the exit code into whatever already stops your pipeline:

python gate.py || { echo "label drift: batch blocked"; exit 1; }

The per-label recall lines matter more than the overall number. An 0.86 overall score that hides rate_limit at 0.4 is worse than an 0.80 score with even coverage, because the batch will keep feeding confident garbage into one branch.

What broke, in the order it broke

1. My golden set was too small and too clean

Twenty examples, two of them labelled schema, meant one miss moved that recall by 50 points. Rebuilding the set to about 80 rows, sampled from the ugly tail rather than from the middle of the distribution, made the gate useful instead of decorative.

2. The same input got different labels on retry

I set temperature to zero and assumed that meant determinism. It did not: when a call failed and I retried it, the label changed for a handful of lines. My ledger stored the second answer without recording that a second answer existed, so the drift was invisible in every aggregate I had.

key = hashlib.sha256(text.encode()).hexdigest()[:16]
ledger.write({"key": key, "state": "inflight", "attempt": n})
label = classify(text)
ledger.write({"key": key, "state": "done", "label": label, "attempt": n})

Writing intent before the call, and the key with it, turned an invisible quality problem into a queryable one. I could finally ask how many keys had more than one distinct label.

3. The free server restarted mid-batch

Nothing dramatic, just a restart that killed the process between the API call and the write. Two hundred lines were classified, paid for in time, and then lost. Checkpointing per line instead of per thousand fixed it, and cost me nothing but a slower disk write.

When this setup is the right call, and when it is not

Situation	Free endpoint + free server	Something with a contract
Throwaway spike, two days, you may delete it	Good fit	Overkill
Fixed label set, golden set under 200 rows	Good fit with a gate	Also fine
Latency-sensitive path, p99 matters	Not the fit	Yes
Regulated data leaving your VPC	No	No, keep it local
Long unattended batch across restarts	Only with per-line checkpoints	Safer

If your task is a live request path with a latency budget, or the data cannot leave your network, this whole article is the wrong shape for you. Use it for offline classification, extraction passes, and experiments where the cost of a wrong answer is a rerun rather than an incident.

What I would repeat, and what I would skip

1. Write the gate before the batch. It took forty minutes and would have saved two days.
2. Keep a golden set that includes the ugly 10% of inputs, not the representative middle.
3. Store an attempt counter next to every label, always.
4. Skip clever chunking on day one. Per-line calls were slow but the failure modes were readable.

Would I run the same spike again? Yes, and I would start from the gate instead of arriving at it, because a schema-valid response is a claim about form and not about meaning.

The harness above is deliberately boring and has no dependency on a specific vendor, so you can point it at any OpenAI-compatible endpoint, including the free model access on offer, and see whether it survives your own golden set.