The Free Tier Was Fine. My Concurrency Was Not.

One request was fast. Fifty at once were not. Same endpoint, same prompt, same model.

So the model was not the problem. My queue was. I stopped timing requests and started timing waits.

Here is the shift that fixed my numbers. Every call has two clocks, not one. There is the time spent waiting for a slot, and the time spent actually working. Most benchmarks only report the sum. The sum hides the queue.

Ask yourself this. When p99 doubles, did the work double? Or did the waiting double? You cannot answer without splitting the clock.

Split the clock before you tune anything

I measure three timestamps per call: sent, first byte, done. Wait equals first byte minus sent. Service equals done minus first byte. That is it.

If wait stays flat while service grows, you have a slow backend. Scale it or shrink the payload. If wait grows while service sits still, you have a queue. More workers will not save you. They will feed it.

That distinction is the whole article. The rest is the harness.

The harness I keep

Zero dependencies. Stdlib only, so it runs anywhere, including on a free server you did not configure.

import json, statistics, time, urllib.request
from concurrent.futures import ThreadPoolExecutor

ENDPOINT = "http://127.0.0.1:8080/v1/chat"  # your proxy, or the free server
BODY = json.dumps({
    "messages": [{"role": "user", "content": "Reply with the word ok."}],
    "max_tokens": 8,
}).encode()

def one_call(_):
    req = urllib.request.Request(
        ENDPOINT, data=BODY, headers={"Content-Type": "application/json"}
    )
    sent = time.perf_counter()
    try:
        with urllib.request.urlopen(req, timeout=120) as r:
            first = time.perf_counter()
            r.read()
    except Exception as e:
        return {"ok": False, "err": type(e).__name__}
    done = time.perf_counter()
    return {"ok": True, "wait": first - sent, "service": done - first}

Note the honest limitation. urlopen returns when headers arrive, not when the first token does. For streaming endpoints, read one chunk instead of the whole body. Otherwise your "service" number is really "time to full response", which is fine, as long as you know it.

Now the sweep. Small levels, modest repeats, one row per level.

def sweep(levels=(1, 2, 4, 8, 16), repeats=20):
    rows = []
    for c in levels:
        t0 = time.perf_counter()
        with ThreadPoolExecutor(max_workers=c) as pool:
            res = list(pool.map(one_call, range(repeats)))
        wall = time.perf_counter() - t0
        ok = [r for r in res if r["ok"]]
        if not ok:
            rows.append({"c": c, "ok": 0, "err": len(res)})
            continue
        waits = sorted(r["wait"] for r in ok)
        servs = sorted(r["service"] for r in ok)
        rows.append({
            "c": c,
            "ok": len(ok),
            "err": len(res) - len(ok),
            "rps": round(len(ok) / wall, 2),
            "wait_p50": round(statistics.median(waits), 3),
            "wait_p95": round(waits[int(0.95 * (len(waits) - 1))], 3),
            "serv_p50": round(statistics.median(servs), 3),
        })
    return rows

Two things make this trustworthy. First, a loopback baseline: point it at a local sleep endpoint first. That number is your client's overhead. Subtract it or at least remember it. Second, fixed payloads. If prompt length drifts between levels, you are measuring your own typing.

The graph I kept is boring and that is why I trust it. X axis is concurrency. Two lines: wait_p95 and serv_p50. I do not care about the absolute values. I care where the lines cross.

Read the crossing, not the average

Here is the decision table taped to my monitor. It is four rows because four is all I can remember under pressure.

What you see	What it means	What to change
wait_p95 under serv_p50	Headroom. Queue is cheap here.	Raise concurrency. Keep sweeping.
wait_p95 roughly equals serv_p50	The knee. Queue cost now matches work.	Stop adding workers. Batch instead.
wait_p95 over 2x serv_p50	Queue-bound. You are paying for waiting.	Split pools, shed load, cache.
Errors before wait moves	Reject-bound, not slow.	Back off. You hit a limit, not a bug.

That last row saved me a weekend once. The endpoint was not slow at all. It was saying no, quickly, and I was reading the empty answers as latency.

Why does the knee matter more than the mean? Because the mean is a lie told by your idle workers. p95 is where your users live during a spike.

Where the free stuff actually helps

The sweep needs two things: tokens and a machine. Running forty calls per level across five levels is not free on a paid API. That is the real friction, not the code.

This is where MonkeyCode's free model access earns its place in the workflow. It lets you run the sweep at small scale without a budget conversation, which means you actually run it. And the free server option lets you put the harness next to the client, so a sleeping laptop or bad wifi stops polluting your wait column. I care about that second part more. Client noise is the most common reason a clean sweep looks like a provider problem.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I will not quote you a token number here. Allowances change, and a stale figure in a blog post is worse than no figure. Check the project page for the current allowance and the terms of the free server before you plan around either.

One rule, and it is not optional. Never point a load harness at an endpoint you lack permission to test. Keep repeats small. Free tiers are shared, and your sweep is someone else's slowdown.

Who should not do this

If you have a contractual p99 with a vendor, a client-side sweep cannot prove anything. You need their capacity numbers and their queueing model, not yours.

If your workload is one big request per minute, concurrency sweeps tell you nothing. Profile the single call instead. And if you cannot make the two-timestamp change in your logging, fix that first. No graph will save a metric you cannot split.

So: what is your knee, and when did you last look at it? Mine moved after a deploy I thought was unrelated. It usually does.

Run the loopback baseline first. Then point it at the real thing. The crossing is the whole story.