A Free Endpoint Isn't Slow — It's Queued: A Little's Law Load Probe

Every week someone tells me free model tiers are unusable. "Too slow," they say. "It just times out."

So I ask one question: how many requests were in flight? Nobody knows. The conversation dies there.

Let's fix that. This is a myth-busting FAQ about concurrency, not about model quality. You get one probe you can run in ten minutes. It replaces vibes with arithmetic.

The setup I'm assuming

You need two things. An endpoint that speaks an OpenAI-style chat API. And a client you control.

Free model access and a free server option are the pieces I'm pointing at here. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I have not verified quotas, model names, hardware, or duration for you. Check those yourself before planning capacity.

Sound fair? Good. Now the myths.

Myth 1 — "More parallel requests means more throughput."

This is the one that burns people. It's true up to a point, then it inverts.

Past saturation, extra concurrency buys latency, not throughput. Your requests sit in a queue you cannot see. From your side it just looks like the server got worse.

The corrected model: concurrency is a budget. You spend it on arrival rate times latency. Overspend, and you buy waiting instead of work.

Little's Law says it plainly: L = λ × W.

• L = requests in the system, on average
• λ = throughput, requests per second
• W = time per request, in seconds

Rearrange it: λ = L / W. Now fix L at the server's ceiling. Throughput can only rise if latency falls. Does latency fall under load? It does not.

Myth 2 — "A 429 is an insult; retry immediately."

No. A 429 is backpressure, and it's the polite version. The rude version is silent queueing.

Immediate retries turn one overload into three. Every retry re-enters the same saturated queue. That's a retry storm, and it's self-inflicted.

The corrected model: a 429 means reduce in-flight, not increase attempts. Backoff without a concurrency cap just spreads the storm across time. It doesn't remove it.

Myth 3 — "Flat throughput means the server is broken."

Flat throughput plus rising latency is the definition of saturation. That's the expected shape, not a bug report.

Ask yourself: did throughput stop climbing exactly when p95 started climbing? If yes, you found the knee. Write the number down. That's your capacity figure for today.

Myth 4 — "You can tune your way out of a capacity limit."

You cannot. You can only shape demand.

Shorter prompts reduce W. Caching reduces arrival rate. Batching reduces request count. None of that raises the ceiling. It just keeps you under it.

That's still worth doing. Just don't call it scaling.

The probe: measure the knee, don't guess it

Here's the artifact. Standard library only, no dependencies, works against any chat-style endpoint.

# probe.py — run: python probe.py
import json, statistics, time, urllib.error, urllib.request
from concurrent.futures import ThreadPoolExecutor

ENDPOINT = "http://127.0.0.1:8000/v1/chat/completions"
HEADERS = {"Content-Type": "application/json"}  # add Authorization if needed
BODY = {"messages": [{"role": "user", "content": "Reply with: ok"}],
        "max_tokens": 8}


def one_call(_):
    data = json.dumps(BODY).encode()
    req = urllib.request.Request(ENDPOINT, data=data, headers=HEADERS)
    start = time.perf_counter()
    try:
        with urllib.request.urlopen(req, timeout=60) as r:
            r.read()
            status = r.status
    except urllib.error.HTTPError as e:
        status = e.code
    except Exception:
        status = 0
    return time.perf_counter() - start, status


def run_level(n_requests, concurrency):
    with ThreadPoolExecutor(max_workers=concurrency) as pool:
        start = time.perf_counter()
        results = list(pool.map(one_call, range(n_requests)))
        wall = time.perf_counter() - start
    lat = sorted(r[0] for r in results)
    ok = sum(1 for r in results if r[1] == 200)
    pick = lambda q: lat[min(len(lat) - 1, int(q * len(lat)))]
    rps = ok / wall
    return {
        "concurrency": concurrency,
        "ok": ok,
        "errors": n_requests - ok,
        "throughput_rps": round(rps, 2),
        "p50_s": round(pick(0.50), 3),
        "p95_s": round(pick(0.95), 3),
        "in_flight_calc": round(rps * statistics.mean(lat), 2),
    }


if __name__ == "__main__":
    for c in (1, 2, 4, 8, 16, 32):
        print(json.dumps(run_level(n_requests=4 * c, concurrency=c)))

That last field, in_flight_calc, is Little's Law running backwards. It tells you how many requests were actually outstanding. Compare it to your thread count. Surprised? You should be.

Reading the output

Four shapes, four stories. Match your rows to this table.

Shape	Throughput	p95	What it means	Do this
Rising	climbs with C	flat	headroom remains	raise C gradually
Knee	flattens	climbing	saturation point	cap C here
Cliff	falls, errors climb	climbing fast	overload and 429s	cut C, add jitter
Flat from C=1	flat	flat	per-request cost dominates	shrink payload, not C

If you landed on the cliff, you now know exactly how much concurrency you were never allowed to use. That's a number. That's useful.

The gate: size your in-flight budget

Here's the rule I use. It comes straight from Little's Law:

MAX_IN_FLIGHT = ceil(target_rps * p95_seconds)

Want 2 rps and your p95 at the knee is 1.5s? Then ceil(2 * 1.5) is 3. Three in flight. Not thirty.

Then wrap calls with a limiter and a sleep budget, so retries can't eat your queue alive:

import asyncio, random

MAX_IN_FLIGHT = 3        # measured, not hoped for
MAX_RETRIES = 4
SLEEP_BUDGET_S = 10.0    # total waiting per task, hard stop

async def call_with_budget(fn, sem, budget):
    async with sem:
        for attempt in range(MAX_RETRIES):
            try:
                return await fn()
            except RateLimited:
                delay = min(8.0, 0.5 * 2 ** attempt) * random.uniform(0.5, 1.5)
                if budget[0] < delay:
                    raise
                budget[0] -= delay
                await asyncio.sleep(delay)
        raise RuntimeError("retries exhausted")

Illlustrative pattern, not a library. Note the jitter. Note the budget. Both exist so a bad minute stays a bad minute.

Limitations, and who should skip this

Be honest about the edges. This method has real ones.

• You are the load. A single laptop probe measures your path, not global capacity.
• Read the terms first. Some free tiers prohibit load testing. Don't be that person.
• Cold starts skew low levels. Run the ramp twice and compare.
• The knee moves. Capacity is a snapshot, not a contract.
• This says nothing about quality. Fast garbage is still garbage.

Don't use this if you need hard SLOs with penalties. Don't use it for high-volume production traffic. Don't use it as a benchmark for someone else's endpoint without permission.

Do use it if you're shipping a small service, a prototype, or an internal tool. That's where queueing surprises hurt most and cost least to fix.

The one-line takeaway

Before you blame the model, count what's in flight. Then apply L = λ × W. Most "slow free tier" stories end right there.

Want to test it yourself? Point the probe at free model access on the free server option, run the ramp, and see where your own knee sits.