Your Free Endpoint Isn't Slow, It's Queued: A Five-Stage Readiness Probe in Pure Python

Free-tier latency is not one number. It's at least four: cold start, steady-state, the concurrency knee, and the error rate you only ever see when you stop sending one request at a time.

Measure only the first and you'll make a bad call — either you'll ship user-facing traffic onto a tier that queues at four concurrent requests, or you'll write off a perfectly usable endpoint because your very first call hit a cold container. Both mistakes cost a week.

So let's build the thing that prevents both: one Python file, five stages, a contract you write before you touch the network, and a verdict you can defend in a review. No third-party packages. About an hour, most of it waiting.

Why a probe instead of a benchmark

A benchmark answers "how fast is this?" A probe answers "is it fast enough for my workload, and what breaks first?" Only the second question survives contact with production.

The loudest AI arguments in my feed lately aren't about model quality at all. They're about access and transparency — who sits between you and the model, and what that middle layer hides. Fair debate. But it's unfalsifiable until you measure your own path to the endpoint. That's all this is: a measurement of your path.

Stage 0 — Write the contract first

If you don't define "fast enough" before you see a single number, you will rationalize whatever you get. So the first artifact is a JSON contract, not code.

{
  "name": "summary-endpoint",
  "prompt": "Summarize in one sentence: free tiers fail politely.",
  "max_tokens": 32,
  "budgets": {
    "single_request_p95_ms": 4000,
    "cold_start_ms": 15000,
    "ladder_p95_ceiling_ms": 12000,
    "error_rate": 0.02
  }
}

Those budgets are placeholders. Replace them with numbers derived from your actual use case — an interactive autocomplete has a very different ceiling than a nightly batch summarizer.

Verification for Stage 0: every budget must be a number you can point at a product requirement for. If you can't, delete it.

Stage 1 — One request, one recorded probe

Config comes from the environment so nothing secret lands in the repo. Set your base URL, key, and model name from your provider's dashboard — and if the endpoint path isn't OpenAI-compatible, MC_PATH is the single line you change.

#!/usr/bin/env python3
"""probe.py - five-stage readiness probe. Pure stdlib."""
import argparse, json, os, statistics, sys, time, urllib.error, urllib.request
from concurrent.futures import ThreadPoolExecutor

BASE = os.environ['MC_BASE_URL'].rstrip('/')
KEY = os.environ['MC_API_KEY']
MODEL = os.environ['MC_MODEL']
PATH = os.environ.get('MC_PATH', '/chat/completions')
LOG = 'probe-records.jsonl'


def call(prompt, max_tokens, timeout=60.0):
    payload = json.dumps({
        'model': MODEL,
        'messages': [{'role': 'user', 'content': prompt}],
        'max_tokens': max_tokens,
    }).encode()
    req = urllib.request.Request(
        BASE + PATH, data=payload,
        headers={'Content-Type': 'application/json',
                 'Authorization': 'Bearer ' + KEY},
    )
    t0 = time.perf_counter()
    rec = {'ok': False, 'status': None, 'retry_after': None, 'tokens': None}
    try:
        with urllib.request.urlopen(req, timeout=timeout) as r:
            body = json.loads(r.read())
            rec['ok'] = True
            rec['status'] = r.status
            rec['tokens'] = (body.get('usage') or {}).get('completion_tokens')
    except urllib.error.HTTPError as e:
        rec['status'] = e.code
        rec['retry_after'] = e.headers.get('Retry-After')
    except Exception as e:            # timeout, DNS, TLS, malformed body
        rec['status'] = type(e).__name__
    rec['ms'] = round((time.perf_counter() - t0) * 1000, 1)
    return rec


def record(stage, **fields):
    line = {'stage': stage, 'ts': round(time.time(), 3), **fields}
    with open(LOG, 'a') as f:
        f.write(json.dumps(line) + '\n')
    print(json.dumps(line))
    return line

One request, one JSON line on disk. Append-only, greppable, and diffable week over week.

Verification for Stage 1: you get "ok": true and a non-null tokens value. If tokens is null, your endpoint isn't returning usage data — note that before you build any cost model on top of it.

Stage 2 — Cold start, which is a waiting problem

Idle the client, then time a single request. Don't guess the idle window; 90 seconds catches most short-lived instance recycling, and you can raise it later.

def stage_cold(cfg, idle):
    print('idling', idle, 's')
    time.sleep(idle)
    rec = call(cfg['prompt'], cfg['max_tokens'])
    budget = cfg['budgets']['cold_start_ms']
    ok = rec['ok'] and rec['ms'] <= budget
    record('cold', ok=ok, ms=rec['ms'], budget=budget, status=rec['status'])
    return ok

Here's the part people skip: a slow cold start is not a reason to reject a tier. It's a reason to send a warm-up request before the first user shows up, or to keep the tier on batch work only.

Verification for Stage 2: run it twice. If the second run is fast, you've confirmed the cold path exists and you know its shape.

Stage 3 — The ladder, where you find the knee

This is the stage that actually decides your architecture. Send 1, 2, 4, then 8 requests at once and watch p50 stay flat while p95 climbs. That divergence is the queue.

def p95(xs):
    xs = sorted(xs)
    return xs[min(len(xs) - 1, int(round(0.95 * (len(xs) - 1))))]


def stage_ladder(cfg, levels):
    for n in levels:
        with ThreadPoolExecutor(max_workers=n) as pool:
            recs = list(pool.map(
                lambda _: call(cfg['prompt'], cfg['max_tokens']), range(n)))
        ms = [r['ms'] for r in recs if r['ok']]
        record('ladder', concurrency=n, sent=len(recs),
               errors=sum(1 for r in recs if not r['ok']),
               p50_ms=round(statistics.median(ms), 1) if ms else None,
               p95_ms=p95(ms) if ms else None)

python probe.py ladder --config contract.json --levels 1,2,4,8

Illustrative output shape — these are placeholders from a made-up run, not a benchmark of anyone's host:

{"stage": "ladder", "concurrency": 1, "errors": 0, "p50_ms": 900, "p95_ms": 950}
{"stage": "ladder", "concurrency": 4, "errors": 0, "p50_ms": 980, "p95_ms": 4100}
{"stage": "ladder", "concurrency": 8, "errors": 3, "p50_ms": 1200, "p95_ms": 9800}

Read that second line again. p50 barely moved; p95 quadrupled. Nothing got slower — requests started waiting their turn. Your in-flight cap belongs just below that point, not above it.

Verification for Stage 3: you can name an integer — "no more than N in flight" — and point at the row that justifies it.

Stage 4 — Push past the knee on purpose

Now run a level you know is too high, and watch your own client. Does it back off, or does it hammer? This stage tests your code, not theirs.

If you see 429s, honor Retry-After when it's present and use exponential backoff with jitter when it isn't. Never retry a 4xx that isn't a rate limit — that's a bug, not congestion.

Verification for Stage 4: errors at 16 concurrent are fine. Errors with no backoff in your logs are not.

Stage 5 — Turn records into a verdict

A decision table beats a vibe. This is the whole payoff of logging JSONL instead of eyeballing a terminal:

Observation	Likely cause	Action
Slow cold start, steady state fine	instance spin-up / cold cache	warm-up ping before user traffic; keep batch work here
p50 flat, p95 climbs with concurrency	shared worker queue	cap in-flight at the knee; queue client-side
Errors only at high concurrency, Retry-After present	rate limit	backoff + jitter; don't retry non-429 4xx
Fast responses, truncated output	max_tokens too low	raise the budget, re-run Stage 1
Everything passes but you re-ran once	small sample	40+ requests before you trust p95

That last row matters. With eight samples, "p95" is basically the maximum. Treat it as a ceiling, never as a percentile.

Where a free key and a free server fit

You can run all five stages against a paid endpoint, and you probably should eventually. But the free tier is exactly where this probe earns its keep, because free capacity is the capacity most likely to share a queue with strangers.

MonkeyCode is one option here: the team advertises free model access plus a free server option, and presents the project as open source. The free allowance was reported to me as 10 million tokens — treat that as a claim to re-check on the project page before you budget against it, because allowances move and my number is already aging. What I can tell you is that the workflow above costs almost nothing to run: a few hundred short requests, a JSONL file, and no dependencies.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Who should not use this approach

Be honest about the fit.

• You need a hard latency SLO for paying users. A probe tells you today's shape; it doesn't give you a contract. Move to a tier with an agreement.
• You can't re-run it weekly. Free capacity shifts. A verdict from six weeks ago is folklore.
• You need a pinned model version. If your output must be reproducible, availability on any free tier is the wrong axis to optimize.
• You haven't read the data-handling terms. I'm making no claim about them here. That's your homework, not this probe's.

The one habit worth keeping

Write the contract, take one measurement per question, and log every probe as a line you can diff. That habit outlives any specific provider — and it turns "the free tier felt slow" into "p95 crossed 12 seconds at eight concurrent requests, so we capped at four."

If you want a zero-cost place to run Stage 0 through Stage 3 and see where your own knee lands, spin up a free key and a free server on MonkeyCode and let the numbers argue for themselves.