Every LLM call in production needs a timeout tuned to the expected output, a retry policy with a hard cap and jitter, an idempotency key on any side effect that follows it, and a circuit breaker for the day the provider is degraded. None of that is AI work. All of it decides whether your feature stays up on a bad afternoon, and it's the layer most AI codebases I inherit are missing.
Why do LLM calls need different defaults from a normal HTTP call?
Because the latency distribution has a long tail and every retry costs real money. A typical internal API answers in 50 milliseconds with a p99 near 300. An LLM call answers in about 2 seconds with a p99 of 25 or more, and a long generation can legitimately run past a minute.
Copy your usual 10-second HTTP timeout onto that and you'll cancel healthy requests, retry them, pay twice, and hit your rate limit sooner. Our defaults, which we then tune per call site, start from the shape of the call:
| Call shape | Total timeout | Idle between tokens | Retries |
|---|---|---|---|
| Short structured call | 30s | 15s | 2 |
| Long generation | 90 to 120s | 15s | 1 |
| Batch job item | 30s | 15s | 2, under a cost ceiling |
Connect timeout is 5 seconds everywhere. The idle timeout is the useful one, because we stream what we can and a stalled stream is a much earlier and cheaper signal than a total timeout.
How many retries, and on what?
Two, at most. Retry on 429s, 5xx responses, connection errors, and timeouts. Use exponential backoff with jitter and honour Retry-After when the provider sends it. Never retry a 400 or a schema-validation failure with the identical request; change something (feed the error back, shorten the input) or give up.
The mistake that taught us the jitter part: a batch job that retried five times with fixed one-second gaps. During a provider incident, a queue of about four thousand items all failed together and all retried together, in lockstep, five times each. We turned a provider slowdown into a self-inflicted rate-limit ban that lasted longer than the original incident. Now every retry policy carries a cost budget too, a per-hour ceiling on retry spend, and the job stops rather than burning through it.
Idempotency: the part people forget
The LLM call itself is usually safe to repeat. What follows it often isn't: send the email, create the ticket, post the comment, charge the card. If your timeout fires after the provider actually finished, and you retry, the model answers twice and the side effect runs twice.
The pattern is borrowed wholesale from payment processing:
- Write the intent before you call the model, as a row with a unique key and status
pending. - Pass that key through as the idempotency key on the side effect itself.
- On any timeout, check whether the work already completed before you retry.
- Mark the row
doneonly once the side effect confirms it ran.
Circuit breakers and what "degraded" looks like
A circuit breaker trips after a threshold of failures inside a window, and while it's open the system stops sending hopeful requests. We start with five failures in sixty seconds per model endpoint. Degraded means you fall back on purpose instead of queueing traffic that has nowhere useful to go.
Fall back to a second provider behind the same interface, a smaller model, a cached answer from an earlier run, or an honest degraded mode ("we'll email you the summary in a few minutes"). Then close the breaker gradually with a trickle of test traffic.
I keep coming back to this because the industry conversation is about which agents are real and which are slop, and I think a large share of what separates agents that last from agents that are slop is this unglamorous layer. At Shanti Infosoft it lives in one shared client wrapper that every AI project imports, so nobody has to remember it and nobody can skip it. It's also the first file we write when a software development team hands us an AI feature that works on their laptop. If your provider's p99 has ever ruined an afternoon, a short call is a cheap way to compare notes.
What does your feature do at 2am when the provider's p99 triples for an hour?
Sagar Jain is the technical co-founder of Shanti Infosoft, where 80+ engineers write the boring layer around the interesting part.
Top comments (0)