One Outage, Four Times the Traffic

The graph that kicked this off: a downstream inventory service wobbled for about two seconds, and our outbound request count to it quadrupled in exactly that window. No user spike. No deploy. The extra traffic was us — my own retry loop, faithfully doing what I told it to do years ago: "try it four times, it's probably transient."

So I rebuilt the incident in miniature. One self-hosted endpoint that returns 503 while a fake outage is active, twenty concurrent callers, and a stopwatch. One process on localhost in a Linux container, .NET 10, delays shrunk so the whole thing fits in seconds — not a lab, I care about the shapes, not the milliseconds.

The loop I used to write

You've seen this loop. Odds are you've written this loop. I've written it more times than I'll admit in public:

for (var attempt = 1; ; attempt++)
{
    var resp = await http.GetAsync("/inventory");
    if (resp.IsSuccessStatusCode) return true;
    if (attempt == 4) return false;
    // no delay — "it's probably transient"
}

Twenty concurrent calls, two-second outage:

== phase 1: hand-rolled retry x4, no delay (outage 2s) ==
client calls :   20    succeeded:   0    failed: 20
server hits  :   80    during outage: 80    after recovery: 0
hit window   : first at      4 ms, last at     33 ms
wall time    : 0.03 s

Read that hit window again. All four attempts, for all twenty callers, landed within 29 milliseconds. The outage lasted two seconds. So my loop generated four times the traffic, aimed every single request at a service that was down, and bought exactly zero extra successes.

That's the quiet flaw in naive retries: they finish before the problem does. A dependency that's down for two seconds might as well be down forever if your entire retry budget burns in 30 milliseconds. And you pay twice — once as wasted work on your side, once as extra load on the exact service that's trying to stand back up.

Same budget, actually spread out

.NET ships a proper answer as a NuGet package: Microsoft.Extensions.Http.Resilience, built on Polly v8. One line on the client registration:

builder.Services.AddHttpClient("backoff", c => c.BaseAddress = new Uri("http://127.0.0.1:5199"))
    .AddStandardResilienceHandler(o =>
    {
        o.Retry.Delay = TimeSpan.FromMilliseconds(500); // production default is 2s; shrunk for the demo
    });

The call site collapses to a single await — the handler owns the retries (three by default), exponential backoff with jitter, per-attempt and total timeouts, and a circuit breaker I'll get to in a minute.

Same twenty calls, same two-second outage:

== phase 2: standard handler, exponential backoff (base 500ms, outage 2s) ==
client calls :   20    succeeded:  13    failed: 7
server hits  :   80    during outage: 67    after recovery: 13
hit window   : first at      1 ms, last at   2741 ms
wall time    : 2.74 s

Here's the part I find genuinely elegant: the request budget is identical. Eighty hits, same as the naive loop. The only thing that changed is where those hits landed in time — spread across 2.7 seconds instead of crammed into 29 milliseconds — and that alone pushed 13 of 20 calls through, because their later attempts outlived the outage. Jitter wobbles the exact split; a back-to-back run gave me 12 of 20, and seven callers still spent their last attempt inside the window. Retries don't help because you try harder. They help when an attempt lands after recovery. Backoff is what buys you that.

Stop knocking on a dead door

Backoff handles the two-second blip. But if the outage runs long, even polite retries are pure cost — every attempt burns your threads and their recovery capacity. That's the circuit breaker's job. The standard handler has one on by default; I tuned it down so 40 demo calls can trip it (the default MinimumThroughput of 100 is sized for real traffic, and rightly so):

.AddStandardResilienceHandler(o =>
{
    o.Retry.Delay = TimeSpan.FromMilliseconds(200);
    o.AttemptTimeout.Timeout = TimeSpan.FromSeconds(2);
    o.CircuitBreaker.FailureRatio = 0.5;
    o.CircuitBreaker.MinimumThroughput = 20;
    o.CircuitBreaker.SamplingDuration = TimeSpan.FromSeconds(10);
    o.CircuitBreaker.BreakDuration = TimeSpan.FromSeconds(2);
});

Forty calls into a six-second outage:

== phase 3: standard handler + circuit breaker (longer outage: 6s) ==
client calls :   40    succeeded:   0    failed: 40
server hits  :   40    during outage: 40    after recovery: 0
hit window   : first at      6 ms, last at     18 ms
wall time    : 0.29 s
failures     : BrokenCircuit (failed fast) x40

probe after recovery : 200 OK — circuit closed itself, no restarts, no config

The naive loop would have thrown 160 requests at that outage. The breaker allowed 40 — the first attempts — saw the failure ratio, opened, and every pending retry failed instantly with BrokenCircuitException instead of queuing up for another knock. Everything resolved in 0.29 seconds. Nothing piled up. The dependency got silence to recover in. And once the break duration passed, a single probe went through half-open, got its 200, and the circuit closed itself.

My opinion, stated as one: a hand-rolled retry loop in a PR should trigger the same reflex as hand-rolled JSON parsing. Not because the loop is hard to write, but because the loop is easy to write badly in ways that only show up during an incident — which is the one time you really don't want surprises.

Where I'd hold back

Retrying a GET is safe. Retrying a POST that charges a card is not "probably transient", it's "possibly twice" — think before wrapping non-idempotent calls, and lean on idempotency keys if you must. Don't copy my demo thresholds to production either; I shrank them so you can watch the breaker trip on localhost, and the real defaults are conservative on purpose. And an open circuit doesn't make failure go away — it makes failure fast. You still owe callers a fallback: a cached value, a sensible default, or an honest 503 of your own. If you don't have one, the breaker just relocates your problem.

Full runnable sample: https://github.com/ssukhpinder/dev-to-code-samples/tree/main/020-httpclient-retry-storm

What's living in your codebase right now — the loop or the handler? If you've survived an actual retry storm (or caused one, no judgment), I'd like to hear how you found it.

— Sukhpinder, still staging outages nobody asked me to