Divide Your Free Tokens by the Acceptance Rate

A free token is not worth a token. It is worth whatever fraction of your calls your checker actually keeps. If six of ten responses fail your own validation, your real budget is not the number on the free-tier page — it is that number times your acceptance rate, and your real latency is the tail of three attempts instead of the mean of one.

So do the division first. Before you promise a quota, a deadline, or a demo, measure two things on your own tasks: how often the free capacity produces output you can keep, and how long the rejects cost you in wall clock. Everything else is arithmetic.

Two multipliers hide behind the word free

Money is the easy part. Free capacity has a price of zero, so it always wins a pure cost comparison, which is exactly why cost comparisons mislead you here. The currency you spend is attempts and seconds.

Attempts burn two budgets at once. They consume the allowance that was supposed to serve other work, and they consume the concurrency slots that make the queue move. Ten teams doing this independently turn a comfortable allowance into a busy afternoon.

Seconds are worse, because they are not symmetric. A single call that takes 900 ms is not thirty calls that take 900 ms when three of them retry — near saturation the retried calls rejoin the same line, so your tail grows faster than your average. The mean stays polite while the p95 quietly breaks your deadline.

A harness small enough to trust

The instrument you need is boring: a fixed task set, a deterministic checker, and a loop that spends attempts until the checker says yes or the attempt budget runs out. No model in the grading step — otherwise you are measuring vibes.

#!/usr/bin/env python3
"""accept_rate.py — tokens and wall clock per *accepted* output.

Run: python3 accept_rate.py --tasks tasks.jsonl --attempts 3 --out results.jsonl
Wire call_model() to the client you are evaluating.
"""
import argparse, json, statistics, time


def call_model(prompt: str) -> dict:
    """Adapter. Replace this body with your client.

    Must return: text, tokens_in, tokens_out
    """
    raise NotImplementedError("wire your client here")


def accepts(task: dict, text: str) -> bool:
    """Deterministic checker. Cheap, strict, and the same for every run."""
    for needle in task.get("must_include", []):
        if needle not in text:
            return False
    if task.get("json"):
        try:
            json.loads(text)
        except json.JSONDecodeError:
            return False
    return len(text.strip()) >= task.get("min_chars", 1)


def run_task(task: dict, attempts: int) -> dict:
    spent = 0
    wall = 0.0
    ok = False
    used = 0
    for used in range(1, attempts + 1):
        t0 = time.monotonic()
        r = call_model(task["prompt"])
        wall += time.monotonic() - t0
        spent += r["tokens_in"] + r["tokens_out"]
        if accepts(task, r["text"]):
            ok = True
            break
    return {"id": task["id"], "ok": ok, "attempts": used,
            "tokens": spent, "wall_s": round(wall, 3)}


def summarize(rows: list) -> dict:
    kept = [r for r in rows if r["ok"]]
    walls = sorted(r["wall_s"] for r in kept)
    p95 = walls[min(len(walls) - 1, int(0.95 * len(walls)))] if walls else None
    spent = sum(r["tokens"] for r in rows)
    return {
        "n": len(rows),
        "acceptance_rate": round(len(kept) / len(rows), 3),
        "tokens_spent": spent,
        "tokens_per_accepted": round(spent / max(1, len(kept)), 1),
        "wall_p50_accepted_s": round(statistics.median(walls), 2) if walls else None,
        "wall_p95_accepted_s": p95,
    }

Sixty tasks is enough to see the shape and short enough to rerun after every prompt change. Keep the task file in git next to the checker. If you cannot review a diff of your eval set, you do not have one.

Read the output as a haircut, not a score

Three numbers matter and the rest are decoration. tokens_per_accepted is your true per-unit price. acceptance_rate is how much of the advertised allowance survives contact with your requirements. wall_p95_accepted_s is the number your users experience whenever the queue is warm.

Then apply the haircut to whatever allowance you were handed, including the operator-stated free allowance of 10 million tokens for the project described below:

allowance = 10_000_000     # stated by the operator; verify the current terms
per_accept = 1_850         # illustrative, from your own run — never from a blog
print(allowance // per_accept)   # ~5,405 accepted outputs before overhead

The arithmetic is the point, not the constant. Substitute your own per_accept and watch the usable number move by a third. That gap is why a gross allowance is a marketing figure and a net allowance is an engineering one.

When free capacity is the wrong bet

Free loses in a small number of recognizable situations, and it loses badly enough that no amount of allowance fixes it.

Situation	Why free capacity loses	Cheaper alternative
Deadline shorter than wall_p95_accepted_s	Retries re-enter the same queue	Pay for the shorter path, or shrink the task
Reject costs a human review	Attempts spend salary, not tokens	Deterministic checker first, then pay
Output must be reproducible for audit	Softmax sampling is not a ledger	Pin seeds or pin a paid endpoint
Sustained throughput above the queue	Tail grows faster than the mean	Committed capacity
The task is the model quality itself	You are evaluating, not producing	Paid endpoint as the reference arm

The last row deserves attention. Free capacity is excellent for evaluation and bad for promises. Run your harness on it, find the acceptance rate, and only then decide what production runs on.

Where MonkeyCode sits in this workflow

MonkeyCode is described by its operator as an open-source project that offers free model access — including the 10-million-token allowance referred to above — and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

In this workflow, both fit as instruments rather than as guarantees. The free model access is the arm you measure first, because zero marginal cost makes it the right place to discover your acceptance rate without watching a meter. The free server option is a reasonable host for the harness itself: clone your repo, create a virtualenv, and run the loop, since the script keeps no state that matters and writes a single JSONL file you can copy off the box.

python3 -m venv .venv && . .venv/bin/activate
git clone <your-harness-repo> && cd harness
python3 accept_rate.py --tasks tasks.jsonl --attempts 3 --out results.jsonl
scp results.jsonl you@laptop:~/runs/$(date +%F)/

Two things to check yourself rather than trust. Read the current terms of the free allowance and the free server before you build a budget on them — availability, quotas, and hardware are operator decisions that change. And if the request, queueing, and retry behavior live in code you can read, read it instead of inferring it from your p95. Inference is what you do when the source is closed.

Limitations, and who should not do this

This method measures one model family against one task set on one day. Provider behavior drifts, so a week-old per_accept is a historical document, not a forecast. The checker is also the weak link: a loose checker inflates the acceptance rate and quietly converts a cost study into a wish.

Skip this entirely if you have a hard latency SLO you cannot renegotiate, if every rejected output requires a human to adjudicate it, or if your workload needs bit-reproducible outputs for audit. In those cases free capacity is not a cheaper option — it is a different product with different terms.

Everyone else should run the harness once before writing the plan. Sixty tasks, one afternoon, and a number you can defend. The allowance on the pricing page is a ceiling; the number your checker returns is a budget, and only one of those survives contact with a deadline.