Myth FAQ: Five Things a Free Token Grant Does Not Prove

My feed is full of model benchmarks right now. It is also full of people tired of them.

So I stopped arguing about leaderboards. I started keeping a ledger instead.

Free model access is the loudest claim in that space. MonkeyCode is an open-source project that offers free model access and a free server option. The operator states a free allowance of 10M tokens.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I will not invent model names, hardware specs, or throughput numbers. I do not have them.

What I can give you is a harness and five myths it can kill.

Why a ledger beats a dashboard

A dashboard shows a number. A number is not evidence.

You need the raw request and response pairs behind it. Status codes, token counts, retry counts, and hashes.

That is the whole artifact. Here it is.

The probe: one file, three prompts

# ledger_probe.py — illustrative harness. No results are reported in this post.
import json, os, time, uuid, hashlib
from pathlib import Path
import httpx

BASE  = os.environ["LLM_BASE_URL"]   # any OpenAI-compatible endpoint
KEY   = os.environ["LLM_API_KEY"]
MODEL = os.environ["MODEL_ID"]       # take this from the project's own list

LEDGER = Path("ledger.jsonl")

PROMPTS = [
    "Reply with the single word: ok",
    "Count the letters in 'strawberry'. Show only the number.",
    "Return strict JSON: {\"n\": 1, \"unit\": \"token\"}",
]

def call(prompt, attempt):
    body = {"model": MODEL, "temperature": 0,
            "messages": [{"role": "user", "content": prompt}]}
    started = time.perf_counter()
    r = httpx.post(f"{BASE}/chat/completions", json=body, timeout=60,
                   headers={"Authorization": f"Bearer {KEY}"})
    elapsed = (time.perf_counter() - started) * 1000

    rec = {
        "probe_id": str(uuid.uuid4()),
        "attempt": attempt,
        "status": r.status_code,
        "latency_ms": round(elapsed, 1),
        "prompt_sha256": hashlib.sha256(prompt.encode()).hexdigest()[:12],
    }

    if r.status_code == 200:
        data = r.json()
        usage = data.get("usage") or {}
        rec["prompt_tokens"] = usage.get("prompt_tokens")
        rec["completion_tokens"] = usage.get("completion_tokens")
        text = data["choices"][0]["message"]["content"]
        rec["completion_sha256"] = hashlib.sha256(text.encode()).hexdigest()[:12]

    with LEDGER.open("a") as fh:
        fh.write(json.dumps(rec) + "\n")
    return rec

for prompt in PROMPTS:
    for attempt in range(1, 4):
        rec = call(prompt, attempt)
        if rec["status"] == 200:
            break
        time.sleep(2 ** attempt)   # free tiers do rate limit. plan for it.

Three prompts, three attempts each. Exponential backoff on failure. Temperature pinned to zero.

Why hashes? Because "it worked" is not a measurement. A hash either repeats or it does not.

The reconcile step

# reconcile.py — reads the ledger you own, not the dashboard you rent.
import json, collections

rows = [json.loads(line) for line in open("ledger.jsonl")]
ok   = [r for r in rows if r["status"] == 200]

print("requests:", len(rows), "ok:", len(ok), "retries:", len(rows) - len(ok))
print("prompt_tokens:    ", sum(r.get("prompt_tokens", 0) for r in ok))
print("completion_tokens:", sum(r.get("completion_tokens", 0) for r in ok))

hashes = collections.Counter(r["completion_sha256"] for r in ok)
print("distinct completions:", len(hashes))
print("most common:", hashes.most_common(3))

Run it. Now you own a receipt.

Myth 1: "10M free tokens means 10M useful tokens"

No. Your prompt overhead eats the grant first.

Every retry re-sends the input. System prompts, tool schemas, and retrieved context all bill on input.

A three-attempt retry on a fat prompt can triple your input spend. On a fixed allowance, retries are not free.

Correct model: useful output is a fraction of the grant. Measure your retry rate before you plan capacity.

Myth 2: "If it answered once, it is deterministic"

One sample tells you almost nothing about variance.

Ask the same question five times. Hash every completion. Count the distinct hashes.

Even at temperature zero, you can see drift across deploys, quantized builds, or routing changes.

Correct model: pin a model version, record the hash, and re-run the probe after every provider change.

Myth 3: "Free means unlimited concurrency"

Free tiers rate limit. That is normal and it is fine.

The dangerous failure mode is silent. A 429 gets logged as "error", then retried forever in a loop.

Log the status code. Stop when you hit a limit. Do not hammer the door you just closed.

• 200: record tokens and hash
• 429: record, back off, cap attempts
• 5xx: record, back off, cap attempts
• other 4xx: fail loudly, do not retry

Correct model: free access is a budget with a governor, not an infinite pipe.

Myth 4: "The free server run is the same as my run"

It is a different machine, a different clock, and possibly a different image.

Environment drift is the quiet bug. Your laptop is on Python 3.12. The box may not be.

So pin what you can control. Lockfiles, container digests, and a dumped environment block.

Write that block into the same ledger, right next to the tokens.

Correct model: the free server is a place to reproduce a result, not a place to assume one.

Myth 5: "The vendor counter is the receipt"

Vendors count what their gateway sees. You count what your client actually sent.

Those two numbers diverge when a proxy retries, when a stream drops, or when a request never lands.

Reconcile both sides. If the gap is more than a few percent, go find out why.

Correct model: two independent counters, one deliberately boring script, and a written note about the gap.

A decision table

Question	Free tier is enough	Move it off
Prototype, low volume	Yes	—
Nightly CI over a fixed prompt set	Yes, with capped attempts	If retries exceed your cap
Customer-facing latency promise	No	Yes
Data residency requirements	No	Yes
Long-context batch jobs	Test it, then measure	If input cost dominates
Anything you cannot re-run	No	Yes

Who should not use this approach

Be honest with yourself before you start.

• Anyone who needs a latency SLA today. Free tiers are not an SLA.
• Teams with data-residency or retention rules. Read those first.
• People who will not read the provider's limit page. You will guess wrong.
• Anyone treating a free grant as permanent capacity. It is a starting point.

A free allowance is a place to learn the cost shape. It is not a capacity plan.

Limitations I will not paper over

I did not verify the current allowance, the model list, or the server specs here. Check the project's own documentation for today's numbers.

Quotas change. A ledger written last month does not know today's limit.

The harness above is illustrative. I am not reporting benchmark results in this post, because I have not run a published one.

Test it before you trust it

Write the ledger first. Argue about models second.

If you want a sandbox to try the harness without wiring up a card, the free model access and the free server option are the relevant parts there. Read the project's own page for current limits before you plan anything.

Then run three prompts. Count the hashes. Reconcile the tokens.

Which myth died first for you?