Workshop: Find Your Endpoint's Concurrency Knee With a 75-Minute Goodput Ramp

Most teams pick a concurrency number for an LLM endpoint by feel, then discover the real ceiling during an incident. The number you did not measure is a guess, and a guess is expensive when your retries start amplifying load. This workshop replaces the guess with a repeatable ramp that plots goodput against offered concurrency, so you can name a defensible ceiling and a shedding rule. Everything below is a teaching outline: exercises, timing, a runnable harness, and a worked example you can rerun against your own gateway.

The core claim this workshop tests

Throughput and goodput are not the same metric, and free or shared endpoints punish the difference quickly. Throughput counts every response that left your client, including the ones that arrived after the caller gave up. Goodput counts only responses that arrived inside the deadline and carried a usable payload, which is the number your product actually experiences. A ramp that tracks both will usually show goodput rising, flattening, then falling while throughput keeps climbing, and that bend is the knee you are looking for.

What you need before the clock starts

• A gateway you control between your client and the model, even a 30-line HTTP wrapper.
• A client machine that is not the machine under test, so client CPU contention does not distort p95.
• A deadline value written down in advance, for example 12 seconds for a short-answer workload.
• A CSV sink and permission to write small files, because the artifact is a table, not a vibe.

This is where a free model endpoint and a free server option genuinely help the method rather than the marketing. Pointing the harness at free model access means you can rehearse the ramp without a billing account, and running the load generator on the free server keeps your laptop off the critical path. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Exercise 1 — Write the harness (15 minutes)

The harness below is a reference implementation; I have not executed it against your endpoint, so treat it as a starting point you must run. It ramps a bounded number of concurrent requests, stamps each result with a deadline verdict, and refuses to hide failures.

# ramp.py — unexecuted reference implementation
import asyncio, csv, json, statistics, time
import httpx

URL = "http://127.0.0.1:8080/v1/chat"   # your gateway, not the vendor URL
DEADLINE_S = 12.0
STEPS = [1, 2, 4, 8, 12, 16, 24, 32]     # offered concurrency, one step at a time
REQS_PER_STEP = 40

async def one(client, sem, idx, out):
    async with sem:
        t0 = time.perf_counter()
        code, ok = None, False
        try:
            r = await client.post(URL,
                json={"prompt": "ping", "max_tokens": 16},
                timeout=DEADLINE_S)
            code = r.status_code
            ok = (r.status_code == 200 and bool(r.text))
        except (httpx.TimeoutException, httpx.TransportError) as e:
            code = type(e).__name__
        out.append({"step": None, "idx": idx, "ok": ok, "code": code,
                    "ms": (time.perf_counter() - t0) * 1000})

async def ramp():
    rows = []
    async with httpx.AsyncClient(http2=False) as client:
        for c in STEPS:
            sem = asyncio.Semaphore(c)
            out = []
            t0 = time.perf_counter()
            await asyncio.gather(*[one(client, sem, i, out) for i in range(REQS_PER_STEP)])
            wall = time.perf_counter() - t0
            for r in out:
                r["step"] = c
            for r in out:
                r["wall_s"] = round(wall, 3)
            rows.extend(out)
            print(c, "concurrency ->", round(wall, 2), "s wall")
            await asyncio.sleep(2)   # let shared queues drain between steps
    return rows

if __name__ == "__main__":
    rows = asyncio.run(ramp())
    with open("ramp.csv", "w", newline="") as f:
        w = csv.DictWriter(f, fieldnames=list(rows[0].keys()))
        w.writeheader(); w.writerows(rows)

Exercise 2 — Add admission control and deadline propagation (15 minutes)

A ramp against an unprotected gateway measures your client, not your system. Add a bounded queue and a deadline header so the server can shed work early instead of finishing requests nobody is waiting for.

# gateway_guard.py — sketch, wire into your own framework
import time, asyncio

class Guard:
    def __init__(self, limit=8, max_wait_s=1.5):
        self.sem = asyncio.Semaphore(limit)
        self.max_wait_s = max_wait_s

    async def run(self, deadline_ms, fn, *a, **kw):
        budget = min(deadline_ms / 1000.0, self.max_wait_s)
        try:
            await asyncio.wait_for(self.sem.acquire(), timeout=budget)
        except asyncio.TimeoutError:
            return {"error": "shed", "reason": "queue_wait_exceeded"}
        try:
            return await fn(*a, **kw)
        finally:
            self.sem.release()

Send X-Deadline-Ms with every request and have the gateway subtract its own elapsed time before calling downstream. When the remaining budget drops below your slowest observed call, shed immediately with a 429 and a Retry-After value. A shed response in 50 ms is cheaper for everyone than a timeout in 12 seconds, and it makes the ramp curve honest.

Exercise 3 — Run the ramp and compute the knee (20 minutes)

The analysis is four lines of pandas, and the formula matters more than the tooling. Goodput per step is successful in-deadline responses divided by wall-clock seconds for that step; the knee is the last step where goodput still improved by your agreed margin over the previous step.

import pandas as pd
df = pd.read_csv("ramp.csv")
df["in_deadline"] = df["ok"] & (df["ms"] <= 12000)
agg = df.groupby("step").agg(
    goodput=("in_deadline", "sum"),
    p95_ms=("ms", lambda s: s.quantile(0.95)),
    wall_s=("wall_s", "first"),
    errors=("ok", lambda s: int((~s).sum())),
)
agg["goodput_rps"] = agg["goodput"] / agg["wall_s"]
agg["gain_pct"] = agg["goodput_rps"].pct_change() * 100
print(agg.round(2))

Use a written rule, not intuition: stop at the last step whose gain is at least 10 percent and whose p95 stays under 60 percent of the deadline. The second condition is the one people skip, and it is the one that prevents a beautiful throughput number from hiding a terrible tail.

Worked example (fill-in template, not measured data)

The table below is deliberately empty of real values; copy it into your card and fill it from your own CSV. Presenting invented numbers here would defeat the point of the workshop.

Offered concurrency	goodput_req/s	p95_ms	errors	verdict
1	fill	fill	0	baseline
4	fill	fill	fill	candidate
8	fill	fill	fill	knee?
16	fill	fill	fill	over

Exercise 4 — Write the shed-versus-queue rule (10 minutes)

Convert the curve into a decision table your on-call engineer can apply without reading the analysis.

Condition at request time	Action	Rationale
Queue depth under limit, budget over 2× slowest p95	queue and serve	capacity exists, patience exists
Queue depth under limit, budget under 1× slowest p95	shed with 429	the response would miss its deadline anyway
Queue depth at limit	shed immediately	protect the requests already admitted
Repeated 429 from upstream	reduce offered concurrency, do not retry faster	retries are load amplification, not recovery

Limitations and who should not use this

A ramp measures today's envelope on a shared endpoint, and it does not transfer to a different provider, region, or hour of the day. Free and shared capacity can change for reasons you cannot observe from the client, so re-run the ramp after any provider-side change and treat old curves as history. Do not run aggressive ramps against endpoints whose terms or rate limits you have not read, and stop the run the moment you see sustained 429 responses. If you need contractual capacity guarantees, multi-region failover, or per-tenant isolation, this workshop is a diagnostic, not a substitute for that engineering.

The 75-minute plan and the artifact you leave with

Minutes	Activity	Output
0-5	Fix the deadline and the workload shape	one written constant
5-20	Exercise 1	ramp.py, ramp.csv
20-35	Exercise 2	guard enabled, X-Deadline-Ms in logs
35-55	Exercise 3	goodput curve, knee candidate
55-65	Exercise 4	shed-versus-queue table
65-75	Fill the capacity card	one page, reviewable

The card is the deliverable: deadline, knee concurrency, p95 at the knee, the shedding rule, and the date of the run. A page with a date beats a dashboard nobody annotated, because it tells the next engineer whether the number still means anything. If you have free model access and a free server option available, run the whole exercise there first and keep your production budget for the changes the curve actually justifies.