DEV Community

Cover image for Retries taught me that most agents retry the wrong errors — lessons from shipping flaky API calls
InApp
InApp

Posted on Originally published at imapp.blogspot.com

Retries taught me that most agents retry the wrong errors — lessons from shipping flaky API calls

I spent a week chasing a weird failure mode where my agent’s outbound calls would sometimes double-send. The upstream was fine most days, then one flaky gateway started returning 502s and my retry loop treated them like a coin flip: retry immediately, three times, no jitter. Under load that turned a small blip into a thundering herd against the very service that was already struggling.

What changed was boring and effective: classify first. 429 and Retry-After mean slow down and respect the header; 5xx means back off exponentially with full jitter; 4xx means stop and surface the error, because retrying a malformed request is just noise. I capped attempts at four, attached an idempotency key to every mutating call, and logged a retry budget per task so I could see when a job was silently spending all its attempts instead of failing loudly.

The measurable win was not “more success,” it was less wasted work. p99 latency dropped, duplicate side effects went to zero, and the error logs finally told the truth: most failures were not transient, they were my payload, my timeout, or my assumptions about someone else’s rate limit.

If you are shipping anything that talks to networks from an agent loop, write the retry policy before you write the happy path. Decide what is idempotent, what must never be replayed, and what the agent should do when the retry budget is gone. The agents that feel reliable are usually just the ones that know when not to retry.

Top comments (0)