Free Tiers Don't Fail Loudly. Your Harness Should.

Cheap model access almost never loses on accuracy. It loses on the tail — the 3% of calls that time out, the one input where the JSON slips, the retry storm that only appears when you fire four requests at once.

So before I trust any free endpoint with real work, I don't write a prompt first. I write a harness that fails loudly, and I write my thresholds down before I run anything.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I'm framing it around MonkeyCode's free model access and free server tier because those are the two things I was handed to work with; the operator states a free token allowance (10M at the time of writing) and a free server tier. Treat both as operator claims and check the current terms yourself. The harness below is endpoint-agnostic — anywhere the OpenAI-compatible chat completions shape works, it runs.

I'm not going to quote pass rates. I don't have them, and making them up would defeat the entire exercise. What I can hand you is the measuring rig, the two numbers that actually change my mind, and the rule I apply to whatever the table says.

Start with the grader, not the model

Every model comparison I've seen go wrong went wrong in the same place: the grading. Someone eyeballs twenty outputs, decides the free model "feels about the same," and ships it. Then a week later the extraction job is silently writing null into a column nobody reads.

So the grader comes first, and it has to be deterministic. No LLM-as-judge for the primary metric — a judge adds its own variance, and if you can't explain the judge's failure mode you've just moved the ambiguity one layer down.

Three task families are enough to start: structured extraction with a strict schema, a small code repair graded by a unit test, and a rewrite task graded by assertions (length, banned phrases, required substrings). Thirty cases, spread across easy and nasty. Freeze them in a file. If a case is ambiguous, delete it — an ambiguous case is a coin flip you'll later misread as a model failure.

The metric that ends arguments: tokens per correct answer

Pass rate alone tells you whether the model can do the job. It doesn't tell you what the job costs. A model that passes 80% while burning three retries per success is more expensive than one that passes 70% and lands it first try.

So I log total_tokens per call, and divide the total spent by the number of correct answers. Tokens per correct answer. That single ratio collapses accuracy, retries, and verbosity into one number you can compare across endpoints, and it's the number I actually defend in a review.

There's a recurring genre of post where a model hits near-perfect accuracy and the author still ships the cheaper one. That's not contrarianism. That's someone reading the right column of the table.

The harness

Here's the runner. It's deliberately small — you should be able to read all of it while your coffee cools.

# harness.py — run N trials of a frozen task suite against any
# OpenAI-compatible endpoint. Records only. No conclusions until the table is full.
import asyncio, json, os, random, time
import httpx

BASE  = os.environ["MC_BASE_URL"]   # e.g. https://<host>/v1
KEY   = os.environ["MC_API_KEY"]
MODEL = os.environ["MC_MODEL"]

TRIALS     = 3      # variance across trials matters more than the mean
CONCURRENCY = 4     # raise this on purpose, then watch retry_rate move
RETRYABLE  = {"http_429", "http_500", "http_502", "http_503", "http_504"}

async def one_call(client, case, trial):
    t0 = time.perf_counter()
    payload = {
        "model": MODEL,
        "messages": [{"role": "user", "content": case["prompt"]}],
        "temperature": 0,                       # pin it, or you're measuring noise
        "max_tokens": case.get("max_tokens", 512),
    }
    r = await client.post(f"{BASE}/chat/completions", json=payload,
                          headers={"Authorization": f"Bearer {KEY}"}, timeout=60.0)
    wall_ms = (time.perf_counter() - t0) * 1000

    if r.status_code != 200:
        return {"case": case["id"], "trial": trial, "ok": False,
                "wall_ms": wall_ms, "tokens": 0, "error": f"http_{r.status_code}"}

    body = r.json()
    text = body["choices"][0]["message"]["content"]
    ok, why = case["check"](text)               # deterministic, returns (bool, str)
    return {"case": case["id"], "trial": trial, "ok": ok, "why": why,
            "wall_ms": wall_ms, "tokens": body.get("usage", {}).get("total_tokens", 0)}

Retries aren't an implementation detail. They're the signal. Count them.

async def call_with_retries(client, case, trial, attempts=3):
    for n in range(attempts):
        rec = await one_call(client, case, trial)
        if rec["ok"] or rec.get("error") not in RETRYABLE:
            rec["retry"] = n
            return rec
        await asyncio.sleep((2 ** n) * 0.5 + random.random() * 0.25)  # jittered backoff
    rec["retry"] = attempts - 1
    return rec

def summarize(rows):
    passed = [r for r in rows if r["ok"]]
    spent  = sum(r["tokens"] for r in rows)
    lat    = sorted(r["wall_ms"] for r in rows)
    pct    = lambda q: round(lat[min(len(lat) - 1, int(q * len(lat)))])
    return {
        "pass_rate":          round(len(passed) / len(rows), 3),
        "tokens_per_correct": round(spent / max(1, len(passed)), 1),
        "p50_ms":             pct(0.50),
        "p95_ms":             pct(0.95),          # the column free tiers fail in
        "retry_rate":         round(sum(r["retry"] for r in rows) / len(rows), 2),
    }

Drive it with a semaphore so concurrency is a knob you control, write every row to JSONL, and keep the raw outputs. You will need them the first time the grader and your eyes disagree.

Read the table, don't read the demo

One good demo proves a model can do the task once. It proves nothing about next Tuesday. Here's the rule I apply, and I set these thresholds before running, because after running I will rationalize anything.

Signal	Threshold I set in advance	What I do when it's crossed
pass_rate	at or above my current baseline	candidate for the cheap path
tokens_per_correct	within 1.5× of baseline	make it the default for batch work
p95_ms	inside my timeout budget	safe behind a synchronous call
retry_rate	flat as concurrency rises	cap concurrency and queue instead
per-case flips across trials	any case that flips	pin harder, or exclude that task

That last row is the one people skip. If the same case passes on trial one and fails on trial three at temperature 0, you don't have a model problem. You have a task too close to the decision boundary, and no amount of prompt polish fixes it.

Where this breaks

Three trials is not statistics. It's a smoke test that catches gross instability and nothing more. If you need confidence intervals, budget for thirty trials per case and accept the token spend.

The grader is the weak link, always. A regex that's too strict scores a correct answer as a failure, and you'll blame the endpoint for your own bug. Audit false negatives by hand for the first run — read every failure, not just the pass rate.

Endpoints drift. Weights get swapped, quantizations change, rate limits move. Your table is a snapshot with a date on it, not a property of the model. And because the whole point here is a free tier, remember the free tier's terms are someone else's to change: verify the allowance and the server offer before you build a pipeline on top of them.

Who should not do this

If your path is user-facing and sub-second, a free tier is the wrong place to save money — go spend the tokens on latency. If you don't have a labeled task suite and won't build one, this harness will give you numbers that feel like answers and aren't. And if a single wrong output costs more than the tokens you saved, the arithmetic is already decided; skip the experiment.

Everyone else: the rig above is maybe ninety lines, and it runs on the same free server tier you're already evaluating. Park it there on a nightly schedule — a harness is only worth something as a series, not a screenshot. Then look at the p95 column before you look at the pass rate. That's where the honesty is.