One successful model request proves only that the request worked once. In production, an integration must withstand 429 responses, provider overload, network timeouts, interrupted streams, and traffic spikes without turning a temporary failure into a retry storm. The practical approach below is to classify errors, limit both elapsed time and attempt count, separate workload types, control concurrency, and define degraded modes in advance.
A demo tests a feature; production tests a system
A prototype usually has one user, a short context, and almost no competing requests. In a production product, several factors combine: one user uploads a large document, a scheduler starts a batch job, a chat service maintains dozens of streaming connections, and all clients retry simultaneously after a brief outage.
AI API quotas are often multidimensional. Providers may limit not only request counts but also input or output token volume. For example, the Gemini API rate-limit documentation describes limits based on requests and input tokens, and exceeding any one dimension is enough to trigger an error. Exact values depend on the model, project, usage tier, and available capacity, so a universal quota table would quickly become outdated.
Reliability here is not a promise that errors will never occur. It is the system’s ability to contain the scope of a failure, preserve important functions, and recover without creating additional load.
Classify the error before choosing an action
An HTTP status alone is insufficient. The client should inspect the machine-readable code in the response body, relevant headers, and the provider’s specific rules. The same 429 status can represent a temporary rate restriction, an exhausted daily quota, or a financial limit. An immediate retry is not appropriate in all of these cases.
| Situation | Retry? | Delay | Limit | Fallback |
|---|---|---|---|---|
| Invalid parameters, unsupported model, or schema error | No, not until the request is corrected | Not applicable | 0 | Validation and an explicit error for the developer or user |
| API key, permission, or billing problem | No, not until the configuration changes | Not applicable | 0 | Stop the task and send an operational alert |
| 429 with a temporary restriction and Retry-After | Yes, if the operation still fits within its deadline | No shorter than Retry-After, plus a small amount of jitter | Within the retry budget | Fast degradation or a queue |
| 429 caused by a daily quota or hard spending limit | Not immediately | Until the quota resets or the limit changes | 0 rapid retries | Defer the work or report that the service is unavailable |
| Temporary 5xx response or model overload | Usually | Retry-After or bounded exponential backoff with jitter | A small number of attempts within the deadline | Another tested configuration or a reduced response |
| Timeout before a result is received | Only if repeating the operation is safe | Backoff followed by a check of the remaining time | Within the overall budget | Queue the task, show a “processing” status, or return an explicit error |
| Interruption after part of a stream has been received | Do not replay automatically | — | 0 hidden retries | Show the partial result and offer an intentional continuation |
| Content block or deterministic rejection | No for the same input | — | 0 | Safely change the workflow or explain the refusal |
This matrix must be adapted to the specific operation. Repeating draft generation usually does not change external state. Repeating an agent call that sends an email, publishes a record, or initiates a payment can perform the action twice. Such operations require an idempotency key, a result journal, and deduplication at the boundary where the side effect occurs.
AI Feed editorial recommendation: the actions, attempt limits, numerical examples, and fallback choices in this table are practical editorial guidance, not thresholds validated by a controlled study. Teams should verify them against their provider contracts, product requirements, and failure tests.
Set a deadline for the entire operation
A timeout on an individual HTTP call does not limit the user’s total wait. Three attempts of 20 seconds each, with pauses between them, can take considerably longer than a minute even though the user has already left the page. Every request should therefore receive an absolute deadline: a point after which no new attempts begin, a queue no longer waits for a response to a closed connection, and child operations are cancelled.
It is useful to distinguish between:
- connection timeout — how long the client waits to establish a connection;
- read timeout — how long the client may receive no data;
- operation deadline — the maximum duration of the complete user action;
- retry budget — the number of attempts and amount of time that may be spent on recovery.
AI Feed editorial example, not a validated study threshold: an interactive assistant could have a total deadline of 12 seconds, one additional attempt only for a temporary error, and at least two seconds reserved for producing a fallback. Background document classification may wait longer, but its job must still have an expiration time: obsolete work should be deleted rather than executed a day later.
Check what the SDK already retries
Some official SDKs automatically retry certain 429 and 5xx responses. The behavior depends on the version and configuration. If an outer service makes three attempts while the SDK makes three attempts inside each one, a single user operation can generate up to nine API calls. This is arithmetic illustrating nested retry multiplication, not a recommended attempt count or a validated threshold.
The OpenAI rate-limits guide recommends limiting both the number of attempts and the total time spent retrying, while accounting for retries built into the SDK. The practical rule is that one component should own the retry policy. Either the application disables automatic SDK retries or includes them in the shared attempt counter and operation deadline.
Retry-After takes precedence over a local formula
If the server provides a valid Retry-After value, the client should not call the service before that interval has elapsed. A small random addition reduces the chance that thousands of clients will wake up at the same moment. If the header is absent or invalid, use bounded exponential backoff with jitter: the pause increases after each failure but has an upper bound.
The algorithm should check before and after sleeping whether the deadline still has time remaining, another attempt is allowed, the circuit breaker is closed, and the user has not cancelled the operation. Infinite retries are unacceptable. Failed requests can themselves consume a per-minute limit, so frequent resubmission can extend an overload event.
Smooth the load before it reaches the provider
Retries can recover from an isolated temporary failure, but they do not regulate the flow of work. Place a bulkhead in front of the AI API to limit the number of calls executing concurrently. This is especially important in fan-out workflows: one file upload can produce hundreds of chunk requests. “Hundreds” is an architecture illustration of possible fan-out, not a universal or validated threshold.
Control not only requests per second but also concurrency, estimated token throughput, and the load attributable to an individual client. Reserve part of the available capacity for interactive actions so that batch indexing cannot displace chat traffic. For confidential or offline workloads, teams can separately evaluate scenarios described in the local AI models guide, but local processing does not eliminate the need for resource management.
A circuit breaker temporarily stops calls to a dependency after a sustained series of failures. It gives the dependency room to recover, but traffic must return cautiously: releasing the entire queue immediately after the breaker closes can create another spike.
Three classes of work require three policies
| Class | Primary concern | AI Feed editorial recommendation |
|---|---|---|
| Interactive chat or suggestion | Short latency | A short deadline, at most one cautious retry, and a fast fallback |
| Batch import | Completeness and controlled throughput | A durable queue, checkpoints, bounded fan-out, and retries for individual parts |
| Background data enrichment | Cost and freshness | Low priority, longer backoff, an expiration time, and deletion of obsolete jobs |
AI Feed editorial qualification: these policies and numbers are practical recommendations, not study-validated thresholds. Production values should be calibrated with load tests, cost constraints, user expectations, and provider-specific behavior.
The queue should store a job identifier, deadline, attempt number, priority class, deduplication key, and a secure reference to the input data. A dead-letter queue is useful for investigation, but it should not become an archive of errors that are retried forever. Queues introduce state, delay, and storage requirements; they are a deliberate tradeoff rather than free reliability.
A streaming response cannot be restarted invisibly
With a streaming API, an error can arrive after several paragraphs have already been displayed. An automatic replay can duplicate text, change the output, or invoke a tool again. Both OpenAI and the Gemini API error documentation address errors delivered inside a stream.
The client should retain stream state and distinguish between “nothing has been displayed” and “part of the result has already been consumed.” In the second case, it is safer to mark the response as incomplete and offer the user a continuation. If the model invokes tools, completed calls should be recorded separately and should not run again without a check.
Degradation must be a product decision
A fallback does not necessarily mean switching to another model. Options include a shorter answer without additional analysis, a cached result with its freshness disclosed, placement in a queue, disabling a nonessential AI feature, or an honest error message.
Switching a model or provider changes the format, quality, latency, cost, and safety properties of the result. It should be tested as a separate product branch. The AI API pricing reference can help evaluate the financial tradeoff, but price does not replace compatibility testing.
Observation: providers document temporary restrictions and overload as normal API operating conditions. Forecast qualification: the frequency of these events in a particular product is unknown until load testing and production operation provide evidence. This technical exposure also does not imply that people will “lose their jobs”: a process’s exposure to failure in an AI component describes a system dependency, not an employment forecast. An analysis of AI’s effects on tasks and occupations requires a separate methodology, such as the AI Jobs Impact Map.
What to measure
- the share of completed user operations, not merely successful HTTP attempts;
- p50, p95, and p99 end-to-end latency, including queueing and retries;
- errors by HTTP status, machine-readable code, model, and provider;
- attempts per operation and total time spent in backoff;
- queue depth and age, including expired and dead-letter jobs;
- input and output tokens, plus an estimate of unreported usage when a stream is interrupted;
- current concurrency, bulkhead rejections, and circuit-breaker state;
- the share of fallbacks, partial streams, and user cancellations.
The percentile labels p50, p95, and p99 are observability categories, not target values or validated thresholds. Pass an internal operation_id, an attempt identifier, and any available provider request ID through the entire chain. Do not record secrets or complete user inputs in logs by default.
Verification methodology
Analysis date: October 19, 2026. This article is based on official OpenAI, Google, and Microsoft documentation available on that date. Its recommendations describe mechanisms rather than fixed quotas.
AI Feed editorial practical checklist — not a validated study protocol or threshold set:
- Artificially lower limits in a test environment.
- Inject a 429 response with Retry-After.
- Inject a 429 response without Retry-After.
- Inject a temporary 503 response.
- Simulate a response that hangs.
- Simulate a network interruption before the first byte.
- Interrupt an SSE stream in the middle of delivery.
- Confirm that the overall operation deadline is respected.
- Confirm that nested retry loops do not occur.
- Confirm that the queue cannot grow without a bound or expiration policy.
- Confirm that interactive traffic retains reserved capacity.
- Confirm that recovery does not release the entire backlog at once.
Run load tests against a production API only with an explicit spending limit, because the test itself can exhaust a quota. The checklist and every practical number above are AI Feed editorial recommendations rather than validated study thresholds. Teams should adapt them to their architecture, provider terms, risk profile, and incident history.
Frequently asked questions
How many times should a request be retried?
There is no universal number. The limit depends on the operation deadline, cost, retry safety, and workload class. An interactive workflow usually needs a very small budget; background jobs may wait longer, but they must also have a finite lifetime. Any example count in this article is an AI Feed editorial recommendation or arithmetic illustration, not a validated study threshold.
Should every 429 response be retried?
No. Determine the machine-readable code and underlying cause. A temporary restriction can permit a delayed retry, while an exhausted daily quota, billing problem, or hard budget limit requires waiting or intervention.
Will a cache solve rate-limit problems?
A cache can reduce average load, but it does not replace throttling. With a cold cache or changing requests, the hit rate can fall sharply, so the system must withstand a large number of cache misses.
When is a queue necessary?
A queue is appropriate when the result does not have to appear in the current HTTP session, the work can be retried safely, and a delay is acceptable. For an interactive response, fast degradation is often preferable to queueing.
What should be implemented first?
AI Feed editorial implementation order: start with an error matrix, an overall deadline, one shared retry budget, and a concurrency limit. Then separate workload classes, add a queue for deferred work, implement observability and a circuit breaker, and test degraded modes. This ordering is practical editorial guidance rather than a study-validated sequence.
Sources and research
- Rate limits — OpenAI API
- Rate limits — Gemini API
- API errors — Gemini API
- Design throttling to improve resilience — Microsoft Azure Well-Architected Framework
AI-assisted article, checked against the primary sources cited above.
Originally published by AI Feed: How to Build a Resilient AI API Integration: Timeouts, Retries, Queues, and Graceful Degradation.
Top comments (0)