Originally published on Loop & Retry — field notes on building LLM agents that survive production.
The retry-budget post answered the HOW: a shared bucket that caps retries as a fraction of throughput. This post answers the WHEN: when should a call fail immediately instead of burning that budget? The answer depends on what fails, who's waiting, and what happens next. It's the decision layer on top of the budget architecture that prevents the kind of bill you get when those decisions are layered without bounds.
Most code doesn't ask those questions. It retries by default — usually some combination of "well, it might work next time" and "the library gives me retry for free so why not." The answer is: because your retry-ready code becomes your customer's timeout, which becomes your platform's cascade.
User-facing calls: cost of latency beats cost of failure
A request from a user interface has a hard deadline. Your browser won't wait longer than ~30 seconds before showing a spinner forever and your customer closes the tab. Within that window you're choosing between two bad outcomes: a snappy error message or a long hang followed by timeout.
For user-facing operations the cost model is simple:
Cost of one retry attempt ≈ seconds_added_to_response_time × 1
Cost of failing now ≈ user_abandonment_rate × minutes_of_lost_engagement × revenue_per_engagement
In practice, most user-facing services find that a 30-second timeout with two quick retries (1s, 3s backoff) is faster to abandon than a 60-second retry-heavy slog. Amazon measured their own checkout and found each 100ms of added latency costs roughly 0.1% of sales. If your retry attempts add 5 seconds to checkout, that's a direct 5% revenue cut — far larger than the probability that a second attempt would succeed at an error that's already firing on the first.
The decision rule: fail fast on user-facing operations unless the error is known-transient and very rare (network timeout, temporary 503). If it's a real error (bad input, rate limit, service misconfigured), retrying won't fix it — it'll just hang your customer.
Where this lives: at the user request handler, not in library code. If your API client retries in userspace, the handler sees the 3rd attempt as a fresh call and can't do anything about it. Retry logic in the wrong layer makes the decision for the caller.
Background jobs: cost of cascade beats cost of wait
A background job has no impatient user; it has a queue and a deadline hours or days away. Here the tradeoff flips: waiting a few seconds to retry is cheap, but failing and re-queueing means the job supervisor will retry the entire task (not just the failed call). That retry at the job level is the expensive one — it re-runs work that might have been halfway done.
A job that retries internally (the call-level budget) can consume partial progress. A job that gives up and re-queues (the job-level retry) starts over. So background jobs should retry longer than user-facing code — but not infinitely.
Cost of call-level retry ≈ seconds_of_work × (already_sunk_cost_fraction)
Cost of job-level re-queue ≈ seconds_of_entire_job × requeue_overhead
If a job is 10 minutes long and a transient error happens at minute 8, one call-level retry that adds 5 seconds costs 5 seconds. Giving up costs 10 minutes of re-queued work (times the overhead). So retry harder.
But "harder" has a ceiling: if the error isn't transient (the service is really down, not just slow), 100 retries over 10 minutes won't help — they'll just pile up in the job queue behind this one, blocking all the others. The decision is not about trying more, it's about trying faster up to a deadline, then escalating.
The pattern: retry the call with a short backoff (1s, 2s, 4s) up to a total budget of 30-60 seconds. If it hasn't worked by then, do NOT give up — escalate to a human alert that says "this batch is stuck, investigate why the service is down." Then the job waits for the alert to be resolved (with exponential backoff, never-ending retry) rather than burning the queue with re-queues.
Where this lives: at the call site within the job, not at the job-queue level. The job-level retry is for when the entire job fails (code bug, bad input). The call-level retry is for transient service issues. They're different concerns.
Continue reading: the rest of this post — including the sections that follow — is on Loop & Retry: read the full article
Top comments (0)