You ship a fix on Monday. By Thursday a teammate posts a number in the channel: "free tier, 87% pass rate." Nobody can reproduce it. The tasks are gone, the endpoint changed, and the run that produced it quietly dropped 40% of its requests on rate limits. The percentage survives in slides long after the conditions that produced it have evaporated.
Free model access makes this worse, not better. Free capacity is the most volatile thing you will ever benchmark: it moves with load, with quotas, with whatever the provider changed this morning. So the number is only worth something if the conditions travel with it.
This is a methodology piece. Below is a small harness that refuses to print an aggregate until its run envelope is complete, two controls that catch a lying grader, and a decision table for deciding whether a number is usable at all.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source coding-agent project that offers free model access and a free server option; those two availability claims are what make the workflow below runnable without provisioning anything. I deliberately do not restate quotas, model names, or hardware here — check the project's current pages for today's terms, because that is exactly the kind of data this article tells you not to trust second-hand.
1. Decide what the number is allowed to claim
Before the loop, write the sentence the result will support. If you cannot write it, you are not ready to run.
Bad: "Model A is better than model B."
Good: "On 40 frozen tasks, with a 4k output budget per task and 3 scheduled attempts each, at 2026-10-17T09:00Z, model A produced 27 verifiable passes and 5 invalid runs; model B produced 24 and 2. This supports a claim about this envelope only."
The second sentence is boring. It is also the only kind that survives a re-run a week later.
Two rules make it work. First, tie the claim to a timestamp and an endpoint, not to a brand. Second, treat invalid runs as a first-class result, because on free capacity they are often the dominant signal.
2. Write the envelope before you write the loop
The envelope is a JSON manifest. Seven fields do most of the work, and one of them is the hash of your task file so the dataset cannot drift out from under the number.
{
"endpoint": "https://<host-from-provider-docs>/v1",
"model_alias": "<exact alias as listed today>",
"token_budget_per_task": 4096,
"max_attempts": 3,
"temperature": 0.0,
"server_class": "free-tier-server, 1 vCPU, 2 GB RAM (as provisioned)",
"concurrency": 1,
"run_date_utc": "2026-10-17T09:00:00Z",
"retry_policy": "retry on 429/5xx/timeout with 2s, 8s backoff; never retry on 4xx",
"task_set_sha256": "<filled in by the harness>",
"invalid_rate_ceiling": 0.10
}
Notice what is absent: no "best", no "fastest", no score. An envelope is a description of a measurement, not a verdict.
The invalid_rate_ceiling is the line you draw in advance. When invalid runs exceed it, the aggregate is not a weak result — it is no result.
3. Build the harness that refuses to aggregate
The core discipline is a gate: if the envelope is incomplete, or too many calls failed, the script exits non-zero and prints why. Never let a denominator shrink silently.
# harness.py — example for a chat-completions style endpoint.
# Run: python harness.py --manifest envelope.json --tasks tasks.jsonl
# Not executed in this article; adapt error handling to your provider's docs.
import argparse, hashlib, json, os, sys, time, urllib.request, urllib.error
REQUIRED = ["endpoint", "model_alias", "token_budget_per_task", "max_attempts",
"temperature", "server_class", "run_date_utc", "retry_policy",
"invalid_rate_ceiling"]
def sha256_file(path):
h = hashlib.sha256()
with open(path, "rb") as f:
for chunk in iter(lambda: f.read(65536), b""):
h.update(chunk)
return h.hexdigest()
def call(m, prompt, key):
body = json.dumps({"model": m["model_alias"],
"messages": [{"role": "user", "content": prompt}],
"temperature": m["temperature"],
"max_tokens": m["token_budget_per_task"]}).encode()
req = urllib.request.Request(m["endpoint"].rstrip("/") + "/chat/completions",
data=body, headers={"Content-Type": "application/json",
"Authorization": f"Bearer {key}"})
t0 = time.time()
try:
with urllib.request.urlopen(req, timeout=180) as r:
payload = json.loads(r.read())
return {"status": 200, "latency_s": time.time() - t0, "payload": payload}
except urllib.error.HTTPError as e:
return {"status": e.code, "latency_s": time.time() - t0, "payload": None}
except Exception as e:
return {"status": 0, "latency_s": time.time() - t0, "payload": None,
"error": repr(e)}
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--manifest", required=True)
ap.add_argument("--tasks", required=True)
ap.add_argument("--out", default="runs.jsonl")
a = ap.parse_args()
m = json.load(open(a.manifest))
missing = [k for k in REQUIRED if m.get(k) in (None, "")]
if missing:
sys.exit(f"REFUSING TO RUN — envelope incomplete: {missing}")
m["task_set_sha256"] = sha256_file(a.tasks)
json.dump(m, open("envelope.frozen.json", "w"), indent=2, sort_keys=True)
key = os.environ["PROVIDER_API_KEY"]
tasks = [json.loads(l) for l in open(a.tasks) if l.strip()]
scheduled = len(tasks) * m["max_attempts"] # denominator, fixed up front
rows, invalid = [], 0
with open(a.out, "w") as out:
for t in tasks:
for attempt in range(1, m["max_attempts"] + 1):
r = call(m, t["prompt"], key)
ok = r["status"] == 200 and r["payload"] is not None
if not ok:
invalid += 1
out.write(json.dumps({"task_id": t["id"], "attempt": attempt,
"status": r["status"], "latency_s": r["latency_s"],
"usage": (r["payload"] or {}).get("usage")}) + "\n")
rows.append({"task_id": t["id"], "ok": ok})
if ok:
break # keep attempts cheap, but keep the denominator fixed
time.sleep(2 if attempt == 1 else 8)
rate = invalid / scheduled
print(f"scheduled={scheduled} invalid={invalid} invalid_rate={rate:.3f}")
if rate > m["invalid_rate_ceiling"]:
sys.exit("NO AGGREGATE — invalid rate above ceiling; the run describes the tier, not the model")
if __name__ == "__main__":
main()
The scheduled = len(tasks) * max_attempts line is the whole point. Your pass rate is passes divided by tasks scheduled, and you report it next to the invalid count. If you divide by successful calls instead, a flaky free tier will look better the more it fails.
4. Add two controls before you trust the grader
A grader that always says "pass" produces a beautiful benchmark. Two cheap controls catch it.
Negative control. Add one task whose correct outcome is failure — for example, "return the SHA-256 of an empty string as 64 zero characters." Nothing should pass it. If your report shows a pass on the negative control, stop: your acceptance check is broken, and every other row is noise.
Replay control. Re-run ten task IDs from the same dataset 24 hours later, same envelope, and compare per-task verdicts. Record the flip rate. On free capacity, a flip rate above roughly 20% means your point estimate should be published as a range, or not at all.
Neither control is statistical sophistication. They are plumbing checks, and most published free-tier numbers skip both.
5. Report five numbers, not one
With runs written to JSONL, aggregate offline so you can re-cut without re-spending quota:
- Pass rate over the frozen denominator, with the scheduled count printed beside it.
- Invalid rate, split by class: 429 / 5xx / timeout / malformed body.
- Tokens per solved task — median over passed tasks only, with the count, because it answers "what does a solution cost me" better than an average over everything.
- p95 latency on valid calls, since the tail is what breaks interactive loops.
- Replay flip rate, from step 4, as a stability annotation on the pass rate.
Publish the task set hash and the frozen envelope.frozen.json alongside these. That is what turns a screenshot into a measurement.
6. A decision table for the result
| Observation | What the number means | Action |
|---|---|---|
| Negative control passes | Grader is broken | Discard run; fix acceptance checks |
| Invalid rate > ceiling | You measured the tier's reliability, not the model | Report invalid rate only; do not publish a pass rate |
| Replay flips > 20% | Envelope is not stable | Publish as interval; increase attempts before comparing |
| Valid runs, stable replay, n ≤ 40 | Directional only | Do not claim differences under ~10 points |
| Valid runs, stable replay, n ≥ 200, two dates | Comparable within this envelope | Safe to cite with hash and date attached |
Read the table before you read the percentage. Most disputes about "who is better" are really disputes about which row of this table the number sits in.
Limitations, and who should skip this
This harness measures behaviour inside one envelope. It says nothing about a model's intrinsic capability, and it will not survive a change to the endpoint, the prompt template, or the token budget. Forty tasks cannot resolve a two-point difference; if that is your question, you need a different instrument, not a bigger table.
Skip this approach if you need SLA-backed numbers for procurement, if you cannot re-run in a week, or if nobody will maintain the task set. A stale harness produces confident garbage.
Also skip it if you only want a demo. Demos are fine — just label them as demos and keep them out of the slide that says "measured".
If you want to try the pipeline without provisioning machines first, MonkeyCode's free model access and free server option are enough to produce a first frozen envelope end to end; read their current terms for what is included today, then write your manifest before you write your loop.
Top comments (0)