Operating an LLM-Backed Service: Budgets, Retries, and Evidence

An LLM-backed endpoint depends on a system whose latency and output size vary with the request. It may also sit behind retrieval, safety checks, and tool calls. Operating it as “just another HTTP request” leaves the difficult behavior invisible until traffic or failures arrive.

A useful runbook begins with budgets and failure semantics, not a dashboard full of model-specific numbers.

Build an end-to-end latency budget

Start with the user-facing objective, then allocate time to stages:

request validation       50 ms
authorization            50 ms
retrieval               400 ms
model time-to-first-byte 2.5 s
stream duration          8.0 s
post-validation         300 ms
response overhead       200 ms

These are example allocations, not universal targets. Measure your workload and choose budgets the product can defend. The important step is assigning ownership: a ten-second endpoint target is not actionable if every dependency assumes it owns all ten seconds.

Track at least two model timings separately:

• time to first output, which shapes perceived responsiveness;
• time to completion, which controls resource use and task completion.

Streaming can improve the first without improving the second.

Use one deadline, propagated downward

Set an overall deadline at the application boundary. Each downstream call receives the remaining time minus a small completion margin. This prevents retrieval, a model call, and a repair attempt from each consuming a full independent timeout.

When the deadline expires:

1. abort downstream work where supported;
2. stop accepting new stream fragments;
3. mark the operation with a typed timeout state;
4. return a user-facing recovery path;
5. retain IDs and timing needed for investigation.

A browser disconnect is a signal, not proof that provider work stopped. Propagate cancellation through the worker and client.

Classify before retrying

Retries should follow the operation’s semantics.

Failure	Typical treatment
Connection failed before request acceptance is known	check idempotency and remaining deadline
Rate limited	respect server guidance; shed or queue load
Provider 5xx	bounded retry with backoff and jitter if budget remains
Invalid request	do not retry unchanged
Generated output fails schema	fail or allow one bounded repair path
Tool action outcome is unknown	reconcile by idempotency key before another action

Never layer retries blindly. If the SDK, service, queue consumer, and gateway each retry three times, one user request can multiply into many provider calls.

Use exponential backoff with jitter to avoid synchronized retry waves. Amazon’s Builders’ Library article on timeouts, retries, and backoff with jitter explains how retries can amplify load and why they need limits.

Make idempotency cross the whole path

For an operation that can create side effects, attach one idempotency key to the application request and persist it with the operation record. Tool executors should use a stable action key derived from that operation and action, not a newly generated key on every retry.

generation operation: gen_123
tool action:           create_ticket
idempotency key:       gen_123:create_ticket:v1

On an ambiguous timeout, query the action record before issuing another call. “The model asked twice” should not mean “the business system executed twice.”

Control cost with admission and limits

Cost controls belong before and during work:

• cap accepted input size by use case;
• cap output size;
• filter retrieval before adding context;
• apply per-user and per-tenant quotas;
• bound tool loops and repair attempts;
• reject or queue work when concurrency is saturated;
• route only evaluated task classes to cheaper models.

Record usage from provider responses when available, but do not make a billing guarantee from an estimate. Reconcile measured usage against provider billing and versioned pricing data.

A useful cost metric has a product denominator, such as cost per completed summary or cost per accepted classification. Cost per request hides retries and unusable output.

Observe operations without collecting everything

For each operation, useful dimensions include:

• operation and trace IDs;
• task type, prompt version, and model alias;
• input and output size or token usage when available;
• queue time, first-output time, and completion time;
• retry count and final status;
• retrieval result count;
• tool name and outcome, without sensitive arguments;
• validation and policy result;
• tenant class or plan, using controlled cardinality.

Do not put raw prompts, generated customer data, or unique user text into metric labels. Logs and traces need explicit redaction and retention rules. The OpenTelemetry generative AI semantic conventions offer common attribute names, while also distinguishing potentially sensitive content fields from ordinary telemetry.

Degrade by capability

Define modes operators can select independently:

normal
  → disable tool writes
  → disable retrieval expansion
  → route to an evaluated fallback model
  → queue non-urgent generation
  → read-only/manual workflow

The safe order depends on the product. A support draft may fall back to a template. An action-taking agent may need tool writes disabled immediately while read-only search remains available.

Keep model alias, prompt version, retrieval configuration, and tool policy independently rollable when possible. A single “AI release” bundle makes it harder to isolate a regression.

Alert on user harm and saturation

Provider error rate matters, but it is not enough. Alert or review on:

• queue age and concurrency saturation;
• deadline-exceeded rate;
• operations without a terminal state;
• duplicate-action prevention events;
• validation failure by prompt/model version;
• fallback activation rate;
• sudden input/output growth;
• cost per successful user outcome;
• severe evaluation or policy failures.

Use distributions for latency and size. Averages hide the long requests that hold connections, consume worker capacity, and shape the worst user experiences.

Rehearse the runbook

Before an incident, test: a slow provider, rate limiting, a stream that stops halfway, malformed structured output, a retrieval outage, a duplicate queue delivery, a tool timeout with an unknown result, and a pricing or quota configuration error.

For each scenario, confirm the user message, operator signal, automated containment, manual switch, and recovery check. The service becomes operable when the team can identify which stage failed, limit additional work, and restore a known configuration without guessing what the model did.