Retry Storms in C#: Exponential Backoff and Jitter with HttpClient

The database blipped for four seconds. The incident lasted forty minutes.

Nothing in the timeline explains the gap except the retry code your own clients are running.

The junior version

while (true)
{
    try
    {
        var response = await client.GetAsync("https://api.internal/orders");
        response.EnsureSuccessStatusCode();
        return await response.Content.ReadAsStringAsync();
    }
    catch
    {
        // transient. it'll come back.
    }
}

// Client log during the incident:
// 14:02:11.418  attempt failed
// 14:02:11.419  attempt failed
// 14:02:11.421  attempt failed
// ...about one attempt per millisecond, per client, across 5,000 clients.

Why it breaks

Start with the part nobody expects: failure is fast. A successful call does DNS, a TLS handshake, a query, and serialisation — tens of milliseconds. A refused connection comes back as a TCP RST in under a millisecond. A 503 from the load balancer never reaches the application at all. So the loop's period is set by the fastest path through the system, and the fastest path is the broken one. The moment the service degrades, every client starts calling it harder than it ever did while healthy.

Now multiply. Normal load is one request per user action. Under failure it is a continuous stream per client, and 5,000 clients pointed at a service sized for a few thousand requests per second will offer it millions. Those attempts are not free just because they fail: each one still costs a socket accept, a TLS handshake, a thread-pool work item, and a slot in the request queue on the server side. The capacity needed to drain the backlog is exactly the capacity being spent rejecting new arrivals.

That is the whole failure mode. At 14:02:15 the original trigger is gone — the failover finished, the slow query completed — and the system stays down anyway, because the retry traffic is now the load. The service has two stable states, and your clients are pinning it in the wrong one. Recovery requires the offered load to drop, which is the one thing a retry loop will never do.

A detour worth taking: a fixed delay is not backoff

This is the fix everyone reaches for first.

catch
{
    Thread.Sleep(1000);   // there. no more spinning.
}

It does stop the spin, and it drops each client from a thousand attempts per second to one. It also blocks a thread-pool thread inside async code, which is its own problem. But the herd survives.

The clients were synchronised by the outage itself. They all failed at 14:02:11.418, so they all wake at 14:02:12.418, and again at 14:02:13.418. The failure event acted as a clock-sync pulse. What the server sees is not a steady 5,000 requests per second — it is all 5,000 arriving at the same instant, once a second, with dead air in between. Peak concurrency is what topples a connection pool, and peak concurrency is unchanged. You have made the graph prettier without moving the number that matters.

The senior version

static async Task<HttpResponseMessage> GetWithRetryAsync(
    HttpClient client, string url, CancellationToken ct)
{
    const int maxAttempts = 5;
    var baseDelay = TimeSpan.FromMilliseconds(200);
    var maxDelay  = TimeSpan.FromSeconds(20);

    for (var attempt = 1; ; attempt++)
    {
        try
        {
            var response = await client.GetAsync(url, ct);
            if (!IsTransient(response.StatusCode) || attempt == maxAttempts)
                return response;
            response.Dispose();
        }
        catch (HttpRequestException) when (attempt < maxAttempts) { }
        catch (TaskCanceledException) when (!ct.IsCancellationRequested
                                            && attempt < maxAttempts) { }

        // Exponential ceiling, then pick uniformly *below* it. That second
        // step is the jitter, and it is the part that scatters the herd.
        var ceiling = Math.Min(maxDelay.TotalMilliseconds,
                               baseDelay.TotalMilliseconds * Math.Pow(2, attempt - 1));
        await Task.Delay(TimeSpan.FromMilliseconds(Random.Shared.NextDouble() * ceiling), ct);
    }
}

static bool IsTransient(HttpStatusCode code) =>
    code == HttpStatusCode.RequestTimeout       // 408
    || code == HttpStatusCode.TooManyRequests   // 429
    || (int)code >= 500;

Exponential growth thins the traffic over time. The uniform draw spreads each wave across the whole interval, so two clients that failed in the same millisecond come back at two random points inside a 200 ms window instead of together — and that window doubles on every attempt after, to a 20-second ceiling. Random.Shared is thread-safe, so you do not need one Random per caller.

Siblings worth knowing:

• Polly v8: new ResiliencePipelineBuilder().AddRetry(new RetryStrategyOptions { BackoffType = DelayBackoffType.Exponential, UseJitter = true }).
• AddStandardResilienceHandler() from Microsoft.Extensions.Http.Resilience, which wires retry, jitter, a circuit breaker and a total timeout into a named HttpClient.
• If the server sends Retry-After (response.Headers.RetryAfter), honour it — then add jitter on top anyway, or every client obeying it wakes in lockstep again.

When backoff isn't enough

Retry is only safe on idempotent operations. A timeout does not tell you the call failed; it tells you the response was lost. The charge may already be posted. GET, PUT and DELETE are idempotent by contract; POST is not. Retrying one needs an idempotency key the server dedupes against, not a smarter delay.

Classify before you retry. A 400, 401 or 404 will fail identically on attempt five. Retrying it burns the caller's latency budget for a guaranteed failure. That is what IsTransient above is for.

Cap elapsed time, not attempt count. Five attempts with a 30-second HTTP timeout each, plus backoff, is over two minutes — long after the browser gave up. Wrap the whole loop in one CancellationTokenSource(TimeSpan.FromSeconds(10)) and let it cut the retries short.

And when the dependency is genuinely down rather than flaky, backoff still has every client politely probing a corpse. That is a circuit breaker's job: stop calling at all, and check back with one request instead of all of them.

The takeaway

Backoff spreads out your retries. Jitter spreads out everyone else's.

Without the second one, you haven't built resilience — you've scheduled a DDoS against yourself, once per second, right on time.

---

I post one of these every day — the same problem solved the way that works and the way that lasts.
One senior tip a week, by email: https://seniorvsjunior.higgsfield.app