Free Tier Is a Budget, Not a Promise: Four Myths and a Capability Card Probe

You find free model access and a free server. You wire up a demo in one afternoon. Then you raise the batch size, and everything crawls.

Was the endpoint broken? Probably not. You built on assumptions you never measured.

That is the whole problem. A free tier is a budget, not a promise. And a budget you never measured is a budget you will overspend.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Per the operator, MonkeyCode offers free model access and a free server option. I will not quote model names, quotas, hardware, or durations here. Those details change, and you should measure your own.

The myths developers repeat

Four claims show up in every "just prototype it" thread. Each one is testable in under an hour.

1. "Free means unlimited retries."
2. "HTTP 200 means the output is usable."
3. "A free server keeps my process alive."
4. "More parallel calls finish the batch faster."

Let's take them one at a time.

Myth 1: Free means unlimited retries

A retry is not free just because the invoice says zero. Every call spends three budgets: requests per window, concurrent in-flight calls, and your own wall-clock time.

The first two are usually hidden. So measure them instead of guessing.

Build a capability card, not a belief

Here is a small probe. It ramps concurrency and records what actually happens.

# probe.py -- a harness you run yourself. Not executed for this article.
import json, time
from concurrent.futures import ThreadPoolExecutor
import requests

ENDPOINT = "https://your-endpoint.example/v1/chat/completions"
HEADERS = {"Authorization": "Bearer " + open(".key").read().strip()}
PROMPT = {
    "model": "the-model-id-from-your-dashboard",
    "messages": [{"role": "user", "content": "Reply with the single word: ok"}],
    "max_tokens": 8,
}

def one_call(_):
    t0 = time.monotonic()
    try:
        r = requests.post(ENDPOINT, headers=HEADERS, json=PROMPT, timeout=30)
        elapsed = time.monotonic() - t0
        try:
            body = r.json()
        except ValueError:
            body = {}
        return {
            "status": r.status_code,
            "seconds": round(elapsed, 3),
            "shape_ok": isinstance(body.get("choices"), list) and bool(body["choices"]),
            "retry_after": r.headers.get("Retry-After"),
        }
    except requests.RequestException as e:
        return {"status": None, "seconds": round(time.monotonic() - t0, 3),
                "shape_ok": False, "error": type(e).__name__}

def ramp(levels):
    rows = []
    for n in levels:
        t0 = time.monotonic()
        with ThreadPoolExecutor(max_workers=n) as pool:
            results = list(pool.map(one_call, range(n)))
        wall = time.monotonic() - t0
        ok = sum(1 for r in results if r["status"] == 200 and r["shape_ok"])
        rows.append({
            "concurrency": n,
            "ok": ok,
            "codes": sorted({str(r["status"]) for r in results}),
            "throughput_per_s": round(ok / wall, 3) if wall else 0.0,
        })
        print(rows[-1])
        time.sleep(1)  # be a polite client while ramping
    return rows

if __name__ == "__main__":
    rows = ramp([1, 2, 4, 8, 16])
    clean = [r["concurrency"] for r in rows if r["ok"] == r["concurrency"]]
    card = {
        "endpoint": ENDPOINT,
        "rows": rows,
        "safe_concurrency": max(clean, default=0),
    }
    json.dump(card, open("card.json", "w"), indent=2)

How do you read the output? Find the last row where ok == concurrency. That row is your knee. Above it, throughput usually flattens or drops.

Record the Retry-After header when a 429 appears. That header is the cheapest rate-limit documentation you will ever get.

Myth 2: HTTP 200 means the output is usable

Status codes describe transport. They say nothing about payload quality.

A truncated body, an empty choices list, or a proxy's HTML error page can all arrive as 200. Your probe already flags the obvious cases with shape_ok.

Corrected model: gate on shape first, then on content. Two cheap checks beat one clever retry.

# gate.py -- shape, then content, then accept. Unexecuted example.
def usable(body):
    choices = body.get("choices")
    if not isinstance(choices, list) or not choices:
        return False
    message = choices[0].get("message") or {}
    content = (message.get("content") or "").strip()
    if not content:
        return False
    return content.lower() != "i cannot help with that"  # too short = suspicious

Is that last line hacky? Slightly. But a length sanity check catches most silent refusals before they reach your database.

Myth 3: A free server keeps my process alive

Free hosts get restarted, suspended, or recycled. Sometimes politely. Sometimes mid-write.

So assume the process can vanish between two lines of your loop. What changes then?

Corrected model: treat the free server as a resumable worker, not a daemon. Write progress after every item, to storage that survives a restart.

# worker.py -- resumable batch loop. Unexecuted example.
import json, os, pathlib

LEDGER = pathlib.Path("done.jsonl")
DONE = set()
if LEDGER.exists():
    DONE = {json.loads(line)["id"] for line in LEDGER.read_text().splitlines() if line.strip()}

def handle(item):
    # your model call goes here
    return {"id": item["id"], "output": "..."}

with LEDGER.open("a") as out:
    for item in load_items():
        if item["id"] in DONE:
            continue
        row = handle(item)
        out.write(json.dumps(row) + "\n")
        out.flush()
        os.fsync(out.fileno())
        DONE.add(item["id"])

The flush plus fsync pair is cheap insurance. On restart, the loop skips completed work instead of re-billing you for it.

Note the distinction. This is a resume mechanism, not a retry policy. Keep those two ideas separate in your head.

Myth 4: More parallel calls finish the batch faster

Past the knee, throughput does not just stall. It can fall, because 429 storms plus backoff cost more than serial calls.

So cap it with the number your card measured.

from threading import Semaphore

CARD = json.load(open("card.json"))
GATE = Semaphore(CARD["safe_concurrency"] or 1)

def guarded(item):
    with GATE:
        return handle(item)

If safe_concurrency comes back as 0, your probe never saw a clean row. Lower the top level and re-run before you ship anything.

The decision table I actually use

Probe result	Reading	Action
Clean at 1, 429 at 2	Single-slot budget	Run serial, cache aggressively
Clean to 4, flat after	Knee at 4	Cap at 4, ignore more cores
200s with empty content	Shape drift	Add the content gate before storage
Process dies mid-run	Ephemeral host	Ledger + resume, never a daemon
Latency varies wildly	Shared capacity	Async batch, not interactive demo

Does every row map to your situation? No. But the table forces a decision instead of a shrug.

Limitations and who should skip this

The probe is a snapshot, not a contract. Free tiers change quotas and routing without a changelog.

It tests one prompt against one endpoint. Another feature, another shape. Re-run after any model switch.

Do not ramp beyond a small ceiling. Levels up to 16 with a one-second pause are enough to find a knee, and gentler on shared capacity.

Skip this whole approach if you need a latency SLO, regulated data handling, or sustained production volume. Buy capacity there. Prototyping economics do not transfer to production guarantees.

A short close

Run the probe before the batch job. Then run it again next month. Measurement is the workflow, not the setup step.

If you want a sandbox for the probe itself, MonkeyCode's free server is a reasonable place to start. Let card.json make the decision for you.