LLM Production Capacity: A SLO-First Load Test for Real Traffic

A production LLM service is ready for load only when it can serve the right requests within an agreed latency and quality boundary at an acceptable cost. Peak tokens per second, a single-request demo, or the vendor's nominal requests-per-minute quota cannot answer that question. Measure the complete path under your traffic shape, distinguish first-token delay from generation time, and count only responses that meet the user-facing service objective. The reusable protocol below is designed for a team choosing hosted inference, self-hosting, or a hybrid route. It extends the production architecture discussion in AI-Due’s guide to deploying an LLM with an explicit pass/fail test and capacity record.

The core metric is goodput: requests completed inside a declared service-level objective (SLO), with valid output and all required safeguards intact. A server that emits many tokens while a queue grows without bound may have impressive throughput but poor goodput. This distinction is supported by the InferBench paper on realistic online LLM load, which argues that saturated offline throughput does not represent bursty interactive traffic. Google’s SRE guidance also warns that request count is a weak capacity proxy when requests have different resource costs, and recommends explicit overload handling (Handling Overload). The practical recommendation is to buy or build capacity against a measured workload and a quality floor, not a headline benchmark.

The decision this protocol resolves

Use this test before approving a production route, a reserved capacity commitment, a self-hosted GPU fleet, a provider migration, or a material increase in users. It answers four separate questions: what load can the route sustain; how does tail latency change as load rises; what is the useful completion rate inside the product’s limits; and what does one accepted task cost after retries, retrieval, moderation, and failed attempts are counted?

It does not decide which model is “best” in general. It does not establish an internet-wide benchmark, prove a provider’s availability, or predict next month’s demand. The protocol compares named configurations for one workload, at one time, with a frozen prompt and software version. A result is valid only for the tested model version, region, API tier, server shape, context distribution, concurrency policy, and output cap. Change one of those and rerun the affected portion.

The production guide contains useful components such as gateways, rate limits, inference servers, retrieval, guardrails, and observability. The missing decision layer is how to test the whole chain without letting high throughput hide slow or invalid answers. This article adds a SLO-bound capacity record: a structured workload, a staged open-loop test, quality and safety gates, a queue stop rule, and a cost calculation. It is a proposed buyer protocol, not a field study.

What the benchmark evidence says—and does not say

The InferBench paper proposes an open-loop load generator, multiple workload types, and reporting throughput alongside time-to-first-token (TTFT), time-per-output-token (TPOT), and latency-bounded goodput. Its abstract reports evaluations of six serving systems on a 13B model, but the public page’s abstract has missing numeric values in its results sentence and says its downloads are not yet available. Therefore, do not repeat the missing values or treat that abstract as independently reproducible results. Its useful contribution here is the measurement design and the distinction between saturation throughput and requests served inside an SLO.

Google’s SRE chapter gives a complementary systems lesson: request cost varies, queues add latency and consume memory, and retry behavior can amplify overload. These observations are not LLM-specific benchmarks. They are operating principles to apply to an LLM route, where input length, output length, retrieval, tool calls, and safety checks make requests heterogeneous. The inference from those two sources is that a single requests-per-second figure is inadequate. The recommendation is to report goodput and tail latency by workload class, along with queue and retry behavior.

Consulting material can help expose cost categories, but it does not replace workload measurement. EY’s discussion of the true cost of AI agents includes more than token charges: platform, infrastructure, governance, workforce, failure, and recovery costs also matter (EY cost model). That is a framing for the cost ledger, not a universal multiplier. Do not turn a consulting estimate into a per-request price without matching its assumptions to your system.

Define the request boundary before testing

Write down where the stopwatch starts and ends. For an interactive assistant, start when the application accepts the user action and stop when the user can act on a complete response. Include network transit, gateway admission, queue time, retrieval, model prefill, token generation, tool calls, validation, and response rendering if they affect perceived wait. If you measure only model-server time, label it as such and do not call it end-to-end latency.

Separate time-to-first-token from time-to-complete. Streaming can make a response feel responsive while the total task remains slow; conversely, a short response may finish quickly but wait a long time before its first token. Record TTFT, total duration, and output length. For systems with tool use, record time spent in each tool and the count of sequential model turns. A request that invokes three tools is not equivalent to a one-shot completion even if both are called one “request” in a provider dashboard.

Choose the SLO from the user task, not from a generic percentile copied from another team. For example, a live agent assist may need a fast first token and a complete recommendation before the human finishes reading the call context. An overnight extraction batch may accept minutes of delay if the queue drains by a defined deadline. State the user impact when the threshold is missed, and state the sample population over which the percentile is calculated. p95 is meaningless without a window, request class, and inclusion rule.

Build a workload that resembles use

Create a frozen set of synthetic or safely anonymized requests. Keep exact prompts, retrieval documents, tool responses, and expected output checks under version control. Stratify by task rather than making every request the same length. Include short questions, long-context summarization, structured extraction, tool-assisted work, refusal or escalation cases, and requests with retrieval. The mix must come from observed product traces where lawful and available; otherwise state that it is a designed workload and not representative production traffic.

Use at least three request classes with explicit input and output limits. For each class record a typical prompt length, a long-context case, a typical answer length, and the maximum allowed answer. Do not infer token length from character count if the tokenizer is available. Keep the content fixed across candidate routes. If you compare providers, use the same semantic task and validation checks, while allowing provider-specific syntax only where required. Save the prompt hash and configuration so a later run can distinguish code changes from sampling variance.

A practical workload manifest has columns for case_id, class, input_tokens, expected_output_schema, safety_case, tool_calls_expected, priority, and quality_check. Include a data-handling note that says whether the set contains personal or confidential information. For a buyer test, use synthetic records by default. A performance test is not permission to send production data to an unapproved endpoint.

Use open-loop arrivals and controlled stages

A closed-loop test sends the next request only after the prior response finishes. That can conceal overload because slow responses automatically reduce the offered load. An open-loop test schedules arrivals independently of response completion, making queue growth visible. The InferBench paper describes an open-loop approach with Poisson and bursty load. You do not need to reproduce its exact statistical model, but you should avoid a test whose offered rate silently falls whenever the service slows.

Run a quiet baseline first, then increase offered load in declared stages. A useful sequence is a low steady rate, a moderate rate, the expected peak, and a short burst above the expected peak. Choose stage duration long enough to observe warm-up and queue behavior; state your duration instead of implying it is universal. Between stages, allow the service and client-side queues to drain and record whether they do. Do not combine cold-start time with steady-state performance unless the product actually experiences cold starts and you report them separately.

Use a fixed random seed for the request schedule if the harness permits it. Record the arrival timestamps, start and end time for each request, status, retry count, token counts, and workload class. Run at least one repeat if a result sits near a gate. If repeatability is poor, investigate variance before averaging; a mean across unlike conditions can hide a failing route. Freeze the provider region, service tier, API version, deployment image, GPU shape, autoscaling policy, and limits for each run.

Measure goodput, not only throughput

Define a request as accepted only if it meets all required conditions: completed before the class-specific deadline; returned a valid response; passed the task’s quality check; respected output schema where required; and did not bypass a safety or authorization control. Then calculate:

accepted_goodput = accepted_requests / measurement_seconds
acceptance_rate = accepted_requests / offered_requests
SLO_miss_rate = requests_missing_latency_or_quality_gate / completed_or_offered_requests

Choose and publish the denominator for each rate. If the system rejects a request immediately because it is overloaded, count that as an SLO failure for the offered workload, even if a separate “completed request” metric excludes it. Report rejections separately so a low latency number cannot be manufactured by dropping slow requests. Report p50, p95, and p99 for TTFT and total duration by class. Use the same percentile method and window across candidates.

A response that arrives within the deadline but is wrong is not goodput. A response that is correct but arrives after the user has abandoned the task is not goodput either. Use deterministic checks where possible: parseable JSON, required fields, exact calculations against known fixtures, and allowed action boundaries. For open-ended answers, use a blinded reviewer rubric with a small, declared sample and report disagreements. Do not claim that a language-model judge is an objective ground truth; it is another measurement instrument with its own error.

Quality and safety thresholds should be hard gates, not weights that latency can compensate for. A route that is fast but leaks restricted information or performs an unauthorized tool call fails. A route that is slightly slower but stays within the user’s accepted deadline may pass. This is why the headline score should not be a weighted average until mandatory thresholds have been applied.

The capacity record

The following table is the reusable artifact. Fill one row per route and workload class; attach the raw result file and software versions. The numerical limits below are placeholders to be set by the product owner, not recommended universal values.

Field	Record for each run	Decision use
Route	Provider/model/version or self-hosted image and hardware	Makes comparison reproducible
Region and tier	Region, service tier, quota, concurrency cap	Explains provider-side differences
Workload	Versioned class mix, input/output distribution, arrival schedule	Defines what was actually tested
TTFT	p50/p95/p99 by class	Detects slow first response
Completion	p50/p95/p99 total duration by class	Measures time until useful result
Goodput	Valid accepted tasks per second inside each SLO	Capacity figure for the buyer decision
Queue	Queue depth and oldest-request age over time	Detects backlog hidden by throughput
Failures	Timeouts, 429s, 5xxs, client errors, malformed output	Shows whether load is being shed or lost
Retry amplification	Initial attempts, retries, duplicate work, final outcomes	Shows load created by recovery behavior
Quality and safety	Schema, accuracy rubric, refusal, permission and policy checks	Hard release gates
Cost	Tokens, tools, retrieval, infra, logging, retry and human review	Cost per accepted task
Caveat	What this run excludes or does not predict	Prevents overgeneralization

A useful release statement looks like this: “Under workload manifest v3, route A served X accepted tasks per second at the defined peak schedule; p95 TTFT and completion remained within the product’s thresholds for classes 1–3; quality and safety gates passed; queue age returned to baseline after the burst; cost was calculated per accepted task. The result applies to model version M, region R, and deployment image D on date T.” If any term is unknown, the claim is not ready for an executive decision.

Estimate the safe operating point

Do not set the production limit at the highest offered rate that passed once. Find the first stage where one of the following begins to deteriorate: tail latency, queue age, rejection rate, error rate, output validity, safety checks, or cost per accepted task. The safe operating point is below that boundary, with a documented buffer that reflects forecast error and recovery time. The buffer is a management choice; it is not a hidden benchmark constant.

The Google SRE chapter cautions that workloads have different resource costs and that excessive queues increase delay and memory use. Apply that by segmenting capacity, not just multiplying a single requests-per-second number. Long-context work may consume substantially different memory and prefill than short prompts. Tool loops can increase the number of model calls per user task. If a route combines classes, run both the aggregate mixture and a class-isolated test so one expensive class cannot starve a critical one unnoticed.

Autoscaling deserves its own test. Measure time to detect load, time to allocate capacity, warm-up delay, and the period during which users still experience the old capacity. A scale-up policy that reacts after the burst is over does not protect the burst. Conversely, scaling to peak permanently can make a self-hosted route uneconomic at low utilization. Record minimum and maximum replicas, scale triggers, cooldown, and whether requests queue during warm-up.

Retries, backpressure, and graceful degradation

Retries can improve completion when a failure is local and transient, but they also add offered load. Google’s overload guidance distinguishes local overload from broad saturation and warns against retries at several layers because they can multiply. Implement one explicit retry owner, a bounded attempt count, exponential backoff with jitter for retryable errors, and a global retry budget. Do not retry validation errors, authorization failures, or requests whose action may already have executed unless the operation is idempotent or reconciled.

Use admission control before expensive work. Reject or queue low-priority work when a capacity budget is exhausted; give the user a clear status and a retry-after estimate only if you can support it. Bound queue size and age. A queue that never rejects converts overload into timeouts and memory pressure. Track whether requests are waiting for a token, a provider quota, a GPU slot, a retrieval store, or a downstream tool; “LLM latency” is too broad to diagnose.

Degraded mode must be an explicit product choice. Examples include returning a shorter answer, skipping nonessential retrieval, or switching a batch task to asynchronous processing. Each degradation needs an allowed-output definition and a quality gate. Do not silently switch to a weaker model if the task is safety-critical, regulated, or depends on a specific capability. Do not remove a safety classifier or permission check to improve latency. If the system cannot meet the minimum acceptable response, return a safe failure or handoff rather than a confident but unverified answer.

For user-facing services, scope the legal analysis to the actual product. Ofcom states that the UK Online Safety Act applies to certain user-to-user and search services, and some chatbot configurations, while some standalone chatbots are outside the described scope; it does not make every LLM API subject to the same duties (Ofcom’s chatbot scope explainer). This source is used here only for the boundary: determine whether the service is in scope before treating a safety control as optional. It is not an inference that every production route has identical UK obligations.

Cost per accepted task

The provider invoice is not the total cost. Calculate the cost of all attempts made for tasks that passed the quality and safety gates, then include non-token components that materially change the decision. EY’s cost framing explicitly covers consumption alongside platform, infrastructure, governance, workforce, failure and recovery costs. Treat that as a checklist of cost buckets rather than a universal ratio; your own invoice, staffing, and architecture determine the amounts.

A basic formula is:

cost_per_accepted_task =
  (provider_or_GPU_cost + retrieval + tools + storage + observability
   + retries + human_review + allocated_operations_cost)
  / accepted_tasks

For a hosted API, record input and output tokens by model, cached-token treatment, tool calls, batch discounts, service tier, and charges for failed attempts if applicable. For self-hosting, include accelerator rental or depreciation, idle capacity, memory, power and cooling where material, orchestration, on-call, upgrades, and redundancy. For hybrid systems, include router costs and the possibility that fallback attempts charge both providers.

Use accepted tasks as the denominator. Dividing by total requests makes a failing route appear cheap if it returns inexpensive errors or invalid output. Also report cost per offered task and the acceptance rate so a decision maker can see both unit economics and quality. Do not compare routes unless they use the same task definition and quality floor. Include the cost of manual correction if people routinely repair model output before it becomes useful.

Worked decision example without invented benchmark results

Suppose a team is deciding whether to keep a hosted model, move a repetitive task to a smaller self-hosted model, or route between them. It has not run the test yet. The honest conclusion is not “hybrid is cheapest” or “self-hosting is faster.” The honest deliverable is a plan: freeze a representative task set, define acceptable completion and safety outcomes, run both routes at the same offered schedule, record resource and quality results, then calculate cost per accepted task.

Before measurement, the team can make a hypothesis: the smaller route may be suitable for short extraction tasks if it passes schema and accuracy checks; a hosted route may remain necessary for long-context cases; and the router may only help if its added latency and duplicate failure paths are outweighed by task acceptance. These are hypotheses, not facts. Mark them as such in the run report and do not place a savings percentage beside them.

The comparison should include a cold-start run if users encounter cold starts, a steady-state run, a burst, and a provider or worker interruption. Record whether the system can tell the user which tasks are delayed, whether work is retried safely, whether queue age is bounded, and whether an operator can stop new admissions. A route that meets the mean but breaks on the burst has not passed a production workload test if bursts are part of the requirement.

How to read an apparent contradiction

A provider may publish a high peak throughput while a research benchmark shows poor SLO-bound goodput under bursty arrivals. Those statements can both be true because one may measure saturation throughput and the other online interactive load. Compare load generator, concurrency, request distribution, hardware, model size, quantization, prompt length, output limit, warm-up, queue policy, and latency SLO before calling one result contradictory.

Similarly, an API provider may advertise automatic scaling while your own route still sees timeouts or rate limits. Provider-level capacity does not guarantee your project quota, network path, gateway, retrieval index, or application deadline. Record the entire path and use the provider’s status and usage views to localize the failing boundary. An LLM endpoint is one dependency in a larger service, not the whole service.

Performance optimizations can trade different properties. Quantization may reduce memory or improve throughput but can alter output quality depending on model and task. Batching may raise aggregate throughput while increasing queue delay for one user. Caching may help repeated inputs but can return stale or context-inappropriate content if the cache key is too broad. Speculative decoding may help some workloads, but a benefit measured in one configuration is not a guarantee for another. Rerun the acceptance set after each optimization and preserve the previous known-good configuration for rollback.

Failure modes that invalidate the result

Closed-loop load masquerading as capacity. When the client waits for each response, offered traffic falls as the service slows. Use open-loop arrivals or explicitly report the closed-loop nature and its limitation.

Averages hiding user pain. An acceptable average can coexist with a failing p95 or p99. Segment by task and report percentiles, sample count, and the exact time window.

Fast rejection counted as fast service. If overload rejections disappear from the latency percentile, users can see a green dashboard while tasks fail. Count them in the offered-task outcome and report them separately.

Output length changing between candidates. A route that produces half as many tokens may look faster. Freeze output budgets and compare task completeness, not tokens alone.

Unbounded queues. Long queues hide overload briefly, then convert it to timeouts, memory pressure, stale work, or a recovery storm. Bound queue age and test saturation behavior.

Retry multiplication. Application, SDK, proxy, and worker retries can stack. Assign retry ownership, set budgets, and verify the attempt ledger under injected failures.

Safety removed in degraded mode. Never improve a performance score by bypassing permissions, review, refusal, or privacy checks. Such a run is not a passing route.

One unusually favorable time window. Shared provider capacity, cache warmth, and network conditions move. Repeat a near-threshold result and note the date, region, version, and service tier.

Misleading “cost per request.” A cheap failed answer is not a cheap accepted task. Include all attempts and labor needed to produce usable work.

Benchmark drift. A prompt, tokenizer, model version, image, server runtime, or quota changes after the baseline. Hash the manifest and rerun gates after material changes.

A small harness contract

The following Python sketch shows the fields a load-test client should capture. The actual HTTP call is deliberately an adapter: provider APIs differ, and a generic script must not imply the same timing or streaming semantics for every endpoint. The harness should use a monotonic clock, record the scheduled arrival separately from actual send time, and emit one JSON line per offered task.

from dataclasses import dataclass, asdict
from time import perf_counter
import json

@dataclass
class Sample:
    case_id: str
    workload_class: str
    scheduled_at: float
    sent_at: float
    first_token_at: float | None
    completed_at: float | None
    status: str
    input_tokens: int | None
    output_tokens: int | None
    attempts: int
    quality_pass: bool | None
    safety_pass: bool | None
    estimated_cost: float | None

async def run_case(case, scheduled_at, call_route):
    sent = perf_counter()
    first = None
    attempts = 0
    try:
        attempts += 1
        result = await call_route(case, on_first_token=lambda: None)
        done = perf_counter()
        sample = Sample(
            case_id=case.id, workload_class=case.kind,
            scheduled_at=scheduled_at, sent_at=sent,
            first_token_at=first, completed_at=done,
            status=result.status, input_tokens=result.input_tokens,
            output_tokens=result.output_tokens, attempts=attempts,
            quality_pass=validate(case, result),
            safety_pass=check_policy(case, result),
            estimated_cost=result.cost,
        )
    except TimeoutError:
        sample = Sample(
            case_id=case.id, workload_class=case.kind,
            scheduled_at=scheduled_at, sent_at=sent,
            first_token_at=first, completed_at=perf_counter(),
            status="timeout", input_tokens=None, output_tokens=None,
            attempts=attempts, quality_pass=False,
            safety_pass=None, estimated_cost=None,
        )
    print(json.dumps(asdict(sample)))
    return sample

This is an instrumentation contract, not a ready-made load generator: call_route, token accounting, validate, policy checks, concurrency limits, open-loop scheduling, and retry policy must be implemented and tested for the chosen provider. In particular, the first_token_at callback must be wired to the actual first streamed content chunk; if the client buffers the full response, it cannot measure TTFT. Do not label a mocked callback as a measured first token.

Original method and evidence limits

This article was prepared on 28 October 2026. The reusable asset is the capacity record, workload manifest, and SLO-bound test protocol above. No live provider, GPU cluster, production workload, or user dataset was measured for this article: original performance sample size is zero systems and zero measured requests. The example is explicitly hypothetical and contains no performance result. The protocol suggests testing multiple workload classes and staged arrival rates, but those are test-design recommendations, not empirical findings.

The GSC selection signal is separate from the article’s technical evidence. On 28 October 2026, the portfolio’s Search Console snapshot listed the AI-Due LLM production architecture page with 243 impressions and an average position of 18.2 over the report’s long aggregation window; the query “production llm api” appeared at position 24.5 with two impressions. This is a low-volume discovery signal, not evidence of demand size, a guarantee of ranking improvement, or proof that this article will rank. The page is used as the source topic because it is already indexable and the query reveals a concrete deployment-intent gap.

Sources have distinct jobs. AI-Due’s page is the primary source for the baseline architecture and performance claims it currently displays. Google SRE supports general overload, queue, and retry principles, not LLM benchmark numbers. InferBench is a research proposal whose abstract informs the metric design; its missing result values and unavailable downloads limit reproducibility from the public page. Ofcom supports only the conditional UK online-safety scope note. EY supports a broader cost-category checklist, not a universal cost multiplier. Where a source does not support a number, this article does not substitute one.

Release decision flow

flowchart TD
  A[Freeze workload and route versions] --> B[Run baseline and staged open-loop arrivals]
  B --> C{Quality and safety gates pass?}
  C -- No --> X[Reject route or fix the failure]
  C -- Yes --> D{Tail latency and queue stay inside SLO?}
  D -- No --> Y[Reduce admission, redesign, or add capacity]
  D -- Yes --> E{Queue drains and retries remain bounded?}
  E -- No --> Z[Fix backpressure and recovery behavior]
  E -- Yes --> F[Calculate cost per accepted task]
  F --> G{Business owner accepts cost and residual risk?}
  G -- No --> H[Change workload or route and rerun]
  G -- Yes --> I[Approve bounded rollout with rollback trigger]

Final checklist

• [ ] The task boundary and user-visible deadline are written down.
• [ ] Workload classes, prompt hashes, token limits, and expected outcomes are frozen.
• [ ] Synthetic or anonymized data is used under an approved data-handling plan.
• [ ] Arrival schedule is independent of response completion, or its closed-loop limitation is explicit.
• [ ] Offered, completed, rejected, timed-out, and accepted tasks are all counted.
• [ ] TTFT, total duration, p50, p95, p99, queue age, and retries are segmented by class.
• [ ] Quality and safety are hard gates; no score can compensate for a failed mandatory check.
• [ ] Queue size, admission control, retry ownership, and graceful-degradation rules are bounded.
• [ ] The cost denominator is accepted tasks and includes retries, tools, retrieval, operations, and human correction where material.
• [ ] The record names route version, region, service tier, image, hardware, date, and exclusions.
• [ ] Near-threshold runs are repeated and the uncertainty is reported.
• [ ] Rollback configuration, owner, alert, and stop trigger are recorded before rollout.

The defensible capacity number is not “tokens per second” in isolation. It is the rate at which a named system completes the tasks your users need, inside their latency and quality boundary, without creating an unsafe shortcut or an unbounded recovery problem. Measure that rate for your own workload, preserve the evidence, and treat any material route change as a reason to revalidate.