Find the Concurrency Where Local Inference Stops Winning

Local inference wins on latency, but only below a concurrency threshold. Above that line, one accelerator becomes a queue, and each new request pays for the ones ahead of it. So stop arguing about which side is "faster" and measure where the two curves cross.

That is the entire method: sweep concurrency, watch p95, find the swap point. Everything below is the harness, the reading rule, and the cases where the cheap remote endpoint genuinely wins.

The claim that gets repeated

Most local benchmarks use exactly one request. The model answers in a few hundred milliseconds and the team declares local faster. Then ten developers share the endpoint, p95 lands in the seconds, and the same team declares local broken.

Both readings are correct at different loads. A single accelerator serves requests roughly serially once it saturates. A hosted fleet spreads them across machines. The number that matters is not latency, it is the concurrency where the two lines exchange places.

The analogy is a supermarket. One open checkout lane is the fastest option when you are the only shopper in the store. It is the worst option at noon.

What to measure, and what to throw away

Four numbers per concurrency level, nothing else: p50 latency, p95 latency, error rate, and completion tokens per second. Pin the prompt, pin max_tokens, set temperature to zero. Warm the model once and discard that call.

Exclude cold start on purpose. Cold start is a separate problem with its own fix. Mixing it into the sweep hides the queueing effect you are trying to observe.

Also count latency where the request actually starts. If your harness queues tasks in a thread pool, the clock must begin inside the worker, not when the task is submitted. Otherwise you measure your own scheduler and learn nothing about the endpoint.

The harness

Labelled clearly: this is an example harness. I have not run it against your machines, and no number in this article comes from it. Your sweep produces your data, which is the point.

# sweep.py - concurrency sweep for any OpenAI-compatible endpoint
import csv, os, sys, time
from concurrent.futures import ThreadPoolExecutor
import requests

BASE   = os.environ["LLM_BASE"].rstrip("/")
MODEL  = os.environ["LLM_MODEL"]
KEY    = os.environ.get("LLM_KEY", "local")
LEVELS = [1, 2, 4, 8, 16]
REQS_PER_LEVEL = 24
PROMPT = "Return the single word ok."


def call(_):
    body = {
        "model": MODEL,
        "max_tokens": 32,
        "temperature": 0,
        "messages": [{"role": "user", "content": PROMPT}],
    }
    t0 = time.perf_counter()
    try:
        r = requests.post(
            f"{BASE}/chat/completions",
            json=body,
            headers={"Authorization": f"Bearer {KEY}"},
            timeout=120,
        )
        ok = r.status_code == 200
        toks = r.json().get("usage", {}).get("completion_tokens", 0) if ok else 0
        return (time.perf_counter() - t0) * 1000, ok, toks
    except Exception:
        return (time.perf_counter() - t0) * 1000, False, 0


def pct(xs, p):
    xs = sorted(xs)
    k = max(0, min(len(xs) - 1, int(round(p / 100 * (len(xs) - 1)))))
    return xs[k]


def main():
    call(0)  # warm-up, discarded
    w = csv.writer(sys.stdout)
    w.writerow(["concurrency", "p50_ms", "p95_ms", "err_rate", "tok_per_s"])
    for n in LEVELS:
        t0 = time.perf_counter()
        with ThreadPoolExecutor(max_workers=n) as pool:
            rows = list(pool.map(call, range(REQS_PER_LEVEL)))
        wall = time.perf_counter() - t0
        lat = [r[0] for r in rows]
        err = sum(1 for r in rows if not r[1]) / len(rows)
        tok = sum(r[2] for r in rows) / wall
        w.writerow([n, round(pct(lat, 50), 1), round(pct(lat, 95), 1),
                    round(err, 3), round(tok, 1)])


if __name__ == "__main__":
    main()

Run it twice, once against each column, with identical prompt and identical max_tokens.

export LLM_BASE=http://127.0.0.1:8080/v1 LLM_MODEL=local-model
python sweep.py > local.csv

export LLM_BASE=https://your-endpoint/v1 LLM_MODEL=remote-model LLM_KEY=...
python sweep.py > remote.csv

Keep REQS_PER_LEVEL at 24 or higher. A p95 computed from eight samples is noise, and you will chase that noise for a week.

Any OpenAI-compatible endpoint fits the second column. MonkeyCode's free model access is one candidate, per the operator, and it also offers a free server option for the case where your laptop should not be the host. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Reading the sweep

The rule is one sentence. The crossover is the smallest concurrency N where remote p95 is lower than local p95 and remote error rate stays at or below one percent. Below that N, the local endpoint keeps the request.

Write the result into the README as a plain statement: "Above six concurrent requests the remote endpoint has the better p95." That line ends most routing arguments before they start.

Watch the gap between p50 and p95 as well. A flat p50 with a climbing p95 means a straggler in the queue, not a bandwidth ceiling. Add a timeout and a fallback before tuning anything else, because tuning a straggler usually moves it rather than removing it.

The offline run

Local-first pays off when the wire is gone, so test that claim instead of assuming it. Two commands are enough.

unshare -n --map-root-user python sweep.py
# or, less polite:
sudo iptables -A OUTPUT -d <remote-ip> -j DROP && python sweep.py; sudo iptables -D OUTPUT -d <remote-ip> -j DROP

Verify two things under isolation. The client fails with a clear error rather than hanging on a socket. And the local path still answers, which requires weights on disk rather than fetched per request.

The offline row is just another column in the same table. If it does not exist, offline support is a claim in a slide, not a property of the system.

Where the free server wins

Three cases, all decidable from the CSV.

Bursts are the first. A CI job or a backfill fires thirty-two requests at once, your laptop queues them, and a hosted endpoint absorbs them. The sweep prints the exact N where that matters.

Model size is the second. If the weights do not fit in memory, local is not slow, it is impossible. That is a capability gap, not a latency trade-off, and no amount of quantization tuning changes the verdict.

Cold hardware is the third. Sleep, thermal throttling, and battery saver all move p95 in ways you cannot control from a config file.

Signal from the sweep	Keep it local	Move it to the free server
Remote p95 wins below N=4	yes	no
Burst of 16 or more	no	yes
Payload contains customer PII	yes	never
Laptop on battery during demos	no	yes

Allowances are the part people over-fit. The operator states free model access and a free server option; confirm the current allowance inside the product before you write it into an architecture document. Quotas move. Your crossing point does not move nearly as fast.

Who should skip this

Do not send traffic to any remote endpoint if the payload cannot leave the machine. That covers customer data, credentials, and most incident text. No latency number outranks that constraint, and the sweep cannot help you there.

Skip the free server if you need a contractual p95. Free tiers are best-effort by nature, and an SLA is a different product.

Skip local if nobody owns the upgrade path. A pinned model still needs patching, and a stale runtime is a security decision made by inaction.

Keep the sweep in the repo

Re-run it when the hardware changes, when the model changes, and once a quarter regardless. Commit both CSVs and the one-sentence crossing point. Reviewers then argue with evidence instead of intuition.

If you want a second column without a payment method, MonkeyCode's free model access and free server option gives you one this afternoon. Run the sweep first, keep whichever side wins at your real concurrency, and write down why you chose it.