Free Tokens Don't Cut the Bill. They Move It.

Free tokens don't cut the bill. They move it.

That is the whole finding, and it is why I rebuilt my evaluation harness before I sent a single prompt through a free model endpoint. When generation costs nothing, the cost does not disappear. It relocates to the part of the pipeline that decides whether the output is any good. And that part is still paid for in seconds, CI minutes, and your own attention.

So the interesting question is not "how many tokens do I get?" It is "what does one accepted output cost me now?" Those are different numbers, and only one of them shows up on a pricing page.

The experiment I actually run

I split the work into three arms. Arm A generates and ships, no gate. Arm B generates once and runs a deterministic oracle against the candidate. Arm C generates, checks, and regenerates a single time on failure before giving up. Everything else is held constant: same task set, same prompt template, same oracle, same machine.

The oracle is the load-bearing piece. It has to be something a machine can settle without judgment — a unit test suite, a JSON schema linter, a compile step, a regex contract on a log line. If you cannot write the oracle, you are not evaluating, you are reading vibes.

The free server matters here for a boring reason. The oracle should run on the same class of infrastructure where the gate will eventually live, not on your laptop, or your verification seconds are fiction. Running the check step on a free server option keeps the measurement honest without renting a bench.

The operator currently lists a free allotment of 10M tokens plus a free server option. Treat that as a claim to verify in their docs before you size anything, not as a spec.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The harness

Below is the skeleton I use. It is unexecuted here and the URLs and model ids are placeholders on purpose — take the real ones from the operator's documentation, because names and limits change and I will not hardcode someone else's quota into your repo.

import argparse, csv, json, subprocess, time, urllib.request
from dataclasses import dataclass, asdict

@dataclass
class Row:
    arm: str
    item: str
    tokens_in: int
    tokens_out: int
    gen_s: float
    verify_s: float
    passed: bool

def generate(base_url, model, token, prompt, timeout=60):
    body = json.dumps({'model': model,
                       'messages': [{'role': 'user', 'content': prompt}],
                       'temperature': 0}).encode()
    req = urllib.request.Request(base_url, data=body,
        headers={'Content-Type': 'application/json',
                 'Authorization': 'Bearer ' + token})
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        payload = json.loads(resp.read())
    text = payload['choices'][0]['message']['content']
    usage = payload.get('usage', {})
    return text, usage.get('prompt_tokens', 0), usage.get('completion_tokens', 0), time.perf_counter() - t0

def oracle(candidate, test_cmd):
    t0 = time.perf_counter()
    proc = subprocess.run(test_cmd, shell=True, input=candidate,
                          text=True, capture_output=True, timeout=120)
    return proc.returncode == 0, time.perf_counter() - t0

Then the loop, which is where the accounting lives:

for item in tasks:
    text, tin, tout, gen_s = generate(base_url, model, token, item['prompt'])
    ok, verify_s = oracle(text, item['test_cmd'])
    attempts = 1
    if not ok and arm == 'C':
        text, tin2, tout2, gen_s2 = generate(base_url, model, token, item['prompt'])
        ok, verify_s2 = oracle(text, item['test_cmd'])
        tin, tout, gen_s, verify_s = tin + tin2, tout + tout2, gen_s + gen_s2, verify_s + verify_s2
        attempts = 2
    rows.append(Row(arm, item['id'], tin, tout, gen_s, verify_s, ok))

Run it with:

python eval_arms.py --arm B --n 30 --out b.csv
python eval_arms.py --arm C --n 30 --out c.csv

The two numbers that decide it

One number is tokens per accepted output: total tokens across the run divided by the count of candidates the oracle passed. The other is verify-seconds per accepted output. Compute both, per arm, and the free tier stops looking free.

If tokens per accepted output is high, the model is producing candidates the oracle keeps throwing away, and the token meter is not your problem — the acceptance rate is. If verify-seconds per accepted output climbs when you move from B to C, the second attempt is buying you candidates at a worse price than the first one did. And if arm A looks great on tokens and terrible on everything downstream, you have just measured the cost of trusting output.

My rule: never publish a single number for a free tier. Republish the run and check whether the median and the spread agree. A wide spread on verify-seconds means the gate has a tail, and a tail is where a batch job turns into a morning.

Where each arm earns its keep

Task shape	Arm	Why
Output is consumed by a human reviewer anyway	A	The reviewer is the oracle; extra checks buy nothing
Output must satisfy a written contract (schema, tests, compile)	B	One deterministic gate captures most of the value
Failure rate is high but failures are cheap to retry	C	A second attempt is cheaper than a human adjudication
Failure is expensive, rare, and silent	B plus a replay set	Retries hide the distribution you need to study
No writable oracle exists	none of these	Go collect labels first

The last two rows are where teams get hurt. Free generation tempts you to skip the oracle entirely and read everything yourself, which is the most expensive gate ever built.

Limitations, and who should walk away

A deterministic oracle only covers tasks that can be settled deterministically. Fluent prose, UI taste, and product judgment do not fit, and mocking up a fake scorer to make the table look complete is worse than leaving the row empty.

My sample sizes here are small and machine-specific, and free tiers change their terms and their hardware without asking me. If you need a contractual SLA, a latency guarantee, or an audit trail you can hand to a regulator, a free endpoint is not the right substrate for the decision path — put the gate on infrastructure you control and use the free tier for the exploratory arms only.

Also skip this workflow if your accept rate is already high. Below a certain failure rate the harness costs more to maintain than it saves, and you should be spending that time on the product.

If you want to try it, pull the endpoint and the model id from the operator's own docs, run arm B first, and trust the verify-seconds column before the token one.