Forty Trials Can Kill a Free Tier. They Can't Crown It.

You do not need two hundred trials to find out that a free model can't hold your JSON schema. You need about forty.

What you cannot do with forty trials — or four hundred — is prove that it can. That is not a test. That is a shadow deployment, and it costs more than the tokens ever will.

Most free-tier evaluations I read are built to certify. Fifty prompts, a pass rate, a green checkmark, ship it. But a pass rate from a small sample is a point estimate with a wide shadow behind it, and the shadow is the only part that matters when your retry loop hits production.

So build the test to reject. Here's the arithmetic that forces it.

The shadow is wider than you think

Suppose forty trials come back at 85% pass. The 95% Wilson interval for that is roughly 0.71 to 0.93. You have not learned that the model is good. You have learned that it is somewhere between "needs a fallback" and "fine" — which is not a decision, it's a coin.

Push the sample size and watch the returns die:

trials	95% interval at p̂ = 0.85	width
20	0.64 – 0.95	31 pts
40	0.71 – 0.93	22 pts
60	0.74 – 0.92	18 pts
100	0.77 – 0.91	14 pts
200	0.79 – 0.89	10 pts

Five times the tokens buys you twice the precision. That is a terrible trade on a metered endpoint and a merely bad one on a free one.

The harness: block-sequential, reject-only

The rule is simple. State the pass rate you require before you run anything. Then stop the moment the upper bound of the interval drops below it. If you run out of blocks without that happening, the answer is not "pass" — the answer is "inconclusive," and you go shadow instead of shipping.

# harness.py — rejection-first free-tier test. Python 3.10+, stdlib only.
import json, math, os, time, urllib.request

Z = 1.96
BLOCK = 10          # trials per decision checkpoint
MAX_TRIALS = 60     # hard stop: past this you are not testing, you are shopping
REQUIRED_RATE = 0.90

def wilson(k, n, z=Z):
    if n == 0:
        return 0.0, 1.0
    p = k / n
    denom = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / denom
    half = (z / denom) * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return max(0.0, centre - half), min(1.0, centre + half)

REQUIRED = {"file", "line", "fix"}

def score(text):
    try:
        obj = json.loads(text)          # strict: fences and prose are failures
    except json.JSONDecodeError:
        return False, "unparseable"
    if set(obj) != REQUIRED:
        return False, "schema"
    if not isinstance(obj.get("line"), int):
        return False, "type"
    return True, "ok"

def call_model(prompt, base_url, key, model):
    body = json.dumps({
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        base_url.rstrip("/") + "/chat/completions",
        data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {key}"},
    )
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=90) as r:
        payload = json.loads(r.read())
    return payload, time.perf_counter() - t0

def run(prompt, base_url, key, model, n_max=MAX_TRIALS):
    k = n = 0
    spent = 0
    failures = {}
    while n < n_max:
        for _ in range(BLOCK):
            payload, _ = call_model(prompt, base_url, key, model)
            n += 1
            spent += payload["usage"]["total_tokens"]
            ok, why = score(payload["choices"][0]["message"]["content"])
            if ok:
                k += 1
            else:
                failures[why] = failures.get(why, 0) + 1
        lo, hi = wilson(k, n)
        if hi < REQUIRED_RATE:
            return {"decision": "REJECT", "k": k, "n": n, "upper": hi,
                    "cost_per_success": spent / max(k, 1), "failures": failures}
    lo, hi = wilson(k, n)
    return {"decision": "INCONCLUSIVE", "k": k, "n": n, "lower": lo,
            "upper": hi, "cost_per_success": spent / max(k, 1),
            "failures": failures}

The endpoint shape above is the OpenAI-compatible chat-completions form. Check the current docs for the base URL and the model ids before you run it — those move, and a stale string will look exactly like a model failure.

Validate the harness before you spend a token

Here is the cheapest step in the whole article and the one almost nobody does. Stand up a fake endpoint with a known pass rate and point the harness at it.

# stub.py — a model with a pass rate you control. STUB_P=0.80 python stub.py
import json, os, random
from http.server import BaseHTTPRequestHandler, HTTPServer

P = float(os.environ.get("STUB_P", "0.80"))
GOOD = {"file": "app.py", "line": 12, "fix": "close the cursor before returning"}

class H(BaseHTTPRequestHandler):
    def do_POST(self):
        self.rfile.read(int(self.headers["Content-Length"]))
        content = json.dumps(GOOD) if random.random() < P else "Sure! Here's the fix:\n```

json\n{}\n

```"
        body = json.dumps({"choices": [{"message": {"content": content}}],
                           "usage": {"total_tokens": 1200}}).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)
    def log_message(self, *a):
        pass

HTTPServer(("127.0.0.1", 8787), H).serve_forever()

Run it twice. With STUB_P=0.80, a working harness should reject the 0.90 claim inside the first few blocks. With STUB_P=0.97, a working harness should never reject, and should end at INCONCLUSIVE. If the 0.97 stub gets rejected, your scorer is buggy and you were about to blame a vendor for your own parser.

I ran it against the stub, not against a live endpoint, on purpose. Pass rates on hosted free models move from week to week; a number I can't re-verify tomorrow isn't evidence, it's a screenshot.

The line is 32 out of 40

Do the work once and you can stop redoing it. At 40 trials, 32 successes is the decision boundary: the upper Wilson bound lands at about 0.895, just under 0.90. Thirty-three successes and you can no longer reject — the interval reaches 0.913.

So the honest reading of a 40-trial run is one sentence long. Thirty-two or fewer passes: kill it, and spend your afternoon on something else. Anything above that: you have failed to reject it, which is not the same as approval.

Why does that matter? Because "failed to reject" is exactly the state in which teams ship, and it's the state in which a 78% model with a friendly first ten outputs looks identical to a 93% model.

Cost per success, not cost per call

Free tokens make people forget the denominator. What you care about is tokens spent per usable output, and retries inflate that fast.

A 1,200-token task at an 0.80 pass rate costs 1,500 tokens per success. At 0.60, it costs 2,000 — a 25% drop in quality is a 33% jump in true cost. Against an allowance of ten million tokens, that is roughly 6,600 successful tasks versus 5,000. Same "free" tier. Same prompt. A third of the work gone.

This is the calculation I actually want before I trust an allowance number, and it's three lines in the harness above: spent / successes. Two numbers, no dashboard.

Keep the model and the host on separate records

This is where a free server will lie to you. A shared host gives you queueing noise, and queueing noise looks exactly like a slow model. Measure it and you'll see the tail separate: a p95/p50 wall-time ratio above about 3 on a cold endpoint usually means you're waiting on someone else's job, not on inference.

So run one small latency-only pass before the accuracy pass, and write the ratio down next to the results. If it's ugly, cap your concurrency at one and re-measure before you conclude anything about the model. Never put shared-host tail latency on a model's permanent record.

When I need a token-hungry harness run and a box to run it on, this is the case where MonkeyCode fits the shape of the problem: free model access for the trial loop, and a free server option for the harness itself. The operator lists a ten-million-token allowance and a free server tier; I'm quoting their numbers, not re-auditing them, and everything above stays valid if you point BASE at any other OpenAI-compatible endpoint instead.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Who should not do this

If the output is judged by a human or by another model on a five-point scale, strict pass/fail scoring collapses and this whole design goes with it — use pairwise comparison instead. If your task is multi-turn agentic work, per-success token accounting gets muddy enough that you should measure cost per completed session, not per reply. And if you genuinely need to distinguish 92% from 97%, forty trials will not do it, four hundred barely will, and the right instrument is a shadow deployment behind a flag, not a test suite.

Everything else — schema adherence, tool-call shape, refusal rate on your own prompts — dies or survives inside about forty trials. The test was never going to tell you yes. It was only ever going to tell you no, and no is the answer you can act on today.