The first production incident in most AI products is a rate limit. The second is duplicate side effects from retrying after a timeout. Both are avoidable with patterns you already know from ordinary API work.
Rate limits: plan for two kinds
Per-minute request caps. These throttle bursts. Handle them with a token bucket and a bounded queue, not with unbounded retries.
Per-minute token caps. Less obvious: a single 100K-token prompt can blow through a token budget that looked generous. Track tokens per minute alongside requests per minute.
Multi-model gateways add a useful escape hatch: when one model family is saturated, route to another family with spare capacity. That only works if your code treats the model as a parameter — which an [OI]-compatible interface gives you by default.
Retries: only safe when idempotent
Retry aggressively on connection errors and 5xx, never on 4xx. But retries create duplicates unless the operation is idempotent. For anything with side effects — sending an email, writing a record, charging a card — attach an idempotency key and dedupe on the receiving side.
For pure inference, retries are usually safe. For agentic flows with tool calls, they are not: a retried tool call can execute twice.
Timeouts are budgets
Set a wall-clock budget per request, not per attempt. If your budget is 20 seconds, three retries of 10 seconds each is a bug, not resilience. Streaming responses need a separate first-token timeout — waiting 20 seconds to start streaming feels broken even if it eventually succeeds.
Observability closes the loop
Log retries per endpoint, tokens per minute, and p95 first-token latency. Those three metrics catch nearly every reliability regression before users report it.
Pricing that publishes input, output and cache rates per model makes capacity planning concrete: KeyoAPI [OI] rates.
Top comments (0)