Measure Your Free AI Tier Before You Trust It: Three Numbers in 20 Minutes

Last month I shipped a tiny CLI that summarizes GitHub issues. It ran fine on a demo repo with three issues. Then a teammate pointed it at a repo with forty issues. The command sat there for ninety seconds and I had no idea why.

I had never measured warm versus cold latency. I had never checked whether the token count in the response matched the token count on my bill. I trusted the docs. The docs were written for someone else's workload.

So I wrote a 60-line probe. It runs in about twenty minutes on a free tier and it tells you three numbers nobody publishes.

Why docs can't answer this

Free tiers are shared, throttled, and re-tuned quietly. A published rate limit is a ceiling, not your throughput. Your p95 latency at 9am on a Tuesday is a fact about your usage, not about their marketing page.

And "free" usually ships with an accounting surface you don't control. If you build a spend alarm on top of a number you never verified, you built the alarm on vibes. Does your budget guard survive a 40% reporting gap? Mine didn't, until I measured it.

The three numbers

1. Warm p95 latency. Not the average. The tail is what makes you add a spinner nobody understands.
2. Token accounting drift. Sum of usage.total_tokens in responses versus your own crude estimate of the same text.
3. Burst failure rate. Five concurrent requests. How many return 429, 5xx, or time out?

That's it. Three numbers, one script, a decision table.

Step 1 — Write the probe

The script below assumes an OpenAI-compatible /chat/completions endpoint. Adjust the request shape to your provider's docs. Nothing here is provider-specific except the env vars.

# probe.py — measure the tier, don't read about it
import os, json, time, statistics, urllib.request, urllib.error
from concurrent.futures import ThreadPoolExecutor

BASE  = os.environ["BASE_URL"].rstrip("/")   # e.g. https://host/v1
MODEL = os.environ["MODEL"]
KEY   = os.environ["API_KEY"]
PROMPT = "Reply with the single word: ok"

def call():
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": PROMPT}],
        "max_tokens": 4,
    }).encode()
    req = urllib.request.Request(
        f"{BASE}/chat/completions", data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {KEY}"})
    t0 = time.perf_counter()
    try:
        with urllib.request.urlopen(req, timeout=120) as r:
            payload = json.loads(r.read())
        return {"ok": True, "secs": time.perf_counter() - t0,
                "usage": payload.get("usage", {})}
    except (urllib.error.HTTPError, urllib.error.URLError, TimeoutError) as e:
        code = getattr(e, "code", None)
        return {"ok": False, "secs": time.perf_counter() - t0, "code": code}

def p95(xs):
    xs = sorted(xs)
    return xs[min(len(xs) - 1, int(round(0.95 * (len(xs) - 1))))]

# 1. one cold-ish call, discarded, then 15 warm calls
call()
warm = [call() for _ in range(15)]
lat  = [r["secs"] for r in warm if r["ok"]]

# 2. accounting drift
reported = sum(r["usage"].get("total_tokens", 0) for r in warm if r["ok"])
local    = len(PROMPT.split()) * len(lat)   # crude word-count floor
drift    = None if not reported else (reported - local) / reported

# 3. burst of 5
with ThreadPoolExecutor(max_workers=5) as pool:
    burst = list(pool.map(lambda _: call(), range(5)))
fails = sum(1 for r in burst if not r["ok"])

print(json.dumps({
    "calls_ok": len(lat),
    "p50_s": round(statistics.median(lat), 2) if lat else None,
    "p95_s": round(p95(lat), 2) if lat else None,
    "reported_tokens": reported,
    "local_estimate": local,
    "drift": None if drift is None else round(drift, 2),
    "burst_failures": f"{fails}/{len(burst)}",
    "burst_codes": [r.get("code") for r in burst if not r["ok"]],
}, indent=2))

Run it with the values from your provider's console:

export BASE_URL="$YOUR_ENDPOINT" MODEL="$YOUR_MODEL" API_KEY="$YOUR_KEY"
python probe.py | tee probe-$(date +%F).json

The output looks roughly like this. These are illustrative values, not a benchmark — yours will differ, and that's the entire point.

{
  "calls_ok": 15,
  "p50_s": 1.7,
  "p95_s": 4.3,
  "reported_tokens": 240,
  "local_estimate": 90,
  "drift": 0.62,
  "burst_failures": "1/5",
  "burst_codes": [429]
}

Step 2 — Read the drift, not the average

That 0.62 drift is the interesting part. My word-count floor ignores system prompts, template overhead, and special tokens. So drift here mostly measures how much hidden text your provider counts that you don't see.

A big drift isn't fraud. It's a warning: your spend alarm needs a fudge factor, or it will fire late. Late alarms on free tiers are how side projects get expensive overnight.

The same logic applies to latency. A p50 of 1.7s with a p95 of 4.3s means one in twenty calls is slow enough that a synchronous CLI feels broken. Now I add a spinner. Before, I guessed.

Step 3 — Turn the numbers into a decision table

Copy this. Fill the middle column with your own measurements. Anything you didn't measure goes in UNKNOWN, and UNKNOWN blocks the deploy.

Number	PASS	UNKNOWN	FAIL
Warm p95 latency	under 3s	not measured	over 10s, or timeouts
Accounting drift	under 10%	10–30%	over 30% with no console total
Burst failure rate	0/5	1/5	2/5 or worse

Three UNKNOWNs is not a soft pass. It's a build blocker with a name attached. Write the owner and the date you'll re-run the probe next to each row.

Step 4 — Re-run it on a schedule

A free tier is not a constant. Ship the probe as a weekly CI job and keep the JSON as an artifact. When p95 doubles, you want a diff, not a mystery.

# .github/workflows/probe.yml (excerpt)
on:
  schedule: [{cron: "0 6 * * 1"}]
jobs:
  probe:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python probe.py | tee probe.json
        env:
          BASE_URL: ${{ secrets.BASE_URL }}
          MODEL: ${{ secrets.MODEL }}
          API_KEY: ${{ secrets.API_KEY }}
      - uses: actions/upload-artifact@v4
        with: {name: probe, path: probe.json}

Where MonkeyCode fits

I ran this probe against MonkeyCode's free model access and its free server option, because that's the environment I actually wanted to depend on. The project is open source, and the operator quotes 10 million free tokens plus a free server. Treat both as time-limited until you've measured them your own way — that's the whole argument of this post.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What the free server changed for me was the boring part. I could run the weekly probe from somewhere that isn't my laptop, and keep the JSON artifacts out of my local disk. That's not exciting. It's exactly the kind of thing that makes a checklist survive past week two.

What this does not tell you

The probe measures a small prompt with max_tokens: 4. That is a latency floor, not your real workload. Long prompts, streaming, and tool calls will behave differently, and often worse.

The drift number is crude by design. A real tokenizer gives you a better estimate; the word-count floor just catches the big gaps fast.

And five concurrent requests is a smoke test, not a load test. If you need to know what happens at fifty, this script will not save you.

Who should not do this

Skip the probe if you already pay for a dedicated endpoint with an SLA. You have better instruments than a for loop.

Skip it if your feature is a one-off script you'll delete Friday. Measuring costs twenty minutes you don't need to spend.

Do not skip it if a stranger's issue report will hit your endpoint while you're asleep.

The checklist I actually keep

• [ ] p95 measured warm, not cold, and written down with a date
• [ ] drift measured against your own estimate, fudge factor documented
• [ ] burst run at 5 concurrent, error codes recorded
• [ ] any UNKNOWN row has an owner and a re-run date
• [ ] probe runs weekly in CI, JSON kept as an artifact

Five lines. One script. If the tier changes next month, you'll know within seven days instead of finding out from a bug report.

If you try this, tell me one thing: which fourth number would you add to the probe for a solo project — cold start on the very first call, or the time it takes to export your data and leave?