Session 5 made the server safe for retries. Today we move to the client side and learn to retry politely.
A retry is a real request — and a client that retries badly can bury a server that was about to recover. In this session we simulate several clients retrying one order the naive way (no waiting) and watch the attempts pile into one tiny window (a retry storm). Then we give them a bounded retry loop with exponential backoff and jitter, and watch the same number of attempts spread calmly over time.
📘 Where we left off in Lesson 5: You made idempotency correct under genuine concurrency. 10 same-key POSTs arriving at the same instant now produce 10 OK responses, 1 unique Order Id, 9
replayed=true, and exactly 1 DB row — no more HTTP 500s for the race losers. Tagged aslesson-05-after. But every "retry" so far was faked by us firing parallel requests. We never built the thing that actually retries in real life — the client. The server is now a safe place to retry into. Today we ask: how should the client retry, so that being safe doesn't accidentally become being slow — or being an outage?🎯 The one idea to take away: Idempotency makes retries safe for correctness. Backoff & jitter make retries safe for system load. Correctness is not the same as scalability. You need both, and they are solved in different places: one on the server (Lesson 5), one on the client (today).
What you'll reinforce and add
🔁 Reinforce — from Lessons 4 & 5:
- Idempotency-Key = one key per business operation
-
replayed=truemeans "I returned the existing result" - DB row count as the authoritative dedup check
- Before/After discipline + the .NET scenario runner
- Reading a console table + saved report
✨ New — added today:
- Seeing a retry as another real HTTP request
- Timeout vs retry vs backoff vs jitter
- The thundering-herd / retry-storm failure mode
- A hand-written exponential-backoff + jitter loop
- Why the SAME key must survive across retries
🎯 Today's goal: Simulate several clients that each retry one order. First let them retry the naive way — no waiting — and watch all the retries pile into one tiny window (a retry storm). Then give them a bounded retry loop with exponential backoff and jitter, reusing the same Idempotency-Key per operation, and watch the same number of attempts spread calmly over time. In both runs the database stays correct (one order per client) — proving the point: the fix here is not about correctness, it's about load.
🧠 Mental Model 1 — A retry is another real HTTP request
The most common beginner mistake is to imagine a "retry" as some lightweight, free thing the framework does in the background. It is not. When a client retries, it opens a real connection and sends a real POST. The server does real work to answer it.
One logical operation: "Create my Koshary + Pizza order"
Attempt 1 ──► POST /api/orders (real request, real CPU, real DB hit)
timeout / no answer in time
Attempt 2 ──► POST /api/orders (ANOTHER real request)
still slow...
Attempt 3 ──► POST /api/orders (ANOTHER real request)
3 attempts = 3 real requests the server must handle.
1 order = what the user actually wanted.
So if 1,000 clients each retry 5 times, that is not 1,000 requests — it is up to 5,000. Retries multiply load. Idempotency makes sure those 5,000 requests still create only 1,000 orders (correctness). It does nothing to reduce the 5,000 (load). That gap is exactly what today is about.
💡 Why does a client even retry? Usually because of a timeout: the request was sent, but no answer came back in time. Here is the cruel part — the client cannot tell the difference between "the server never got it" and "the server did the work but the reply got lost." So a careful client must retry to be safe… which is only OK because the server is idempotent. The two halves depend on each other.
🧠 Mental Model 2 — Timeout vs Retry vs Backoff vs Jitter
Four words that beginners often blur together. They are four different decisions:
TIMEOUT → "How long do I wait for one answer before I give up on it?"
e.g. wait 2s; if no response, treat this attempt as failed.
RETRY → "After a failed attempt, do I try again, and how many times?"
e.g. try at most 5 times, then surface an error to the user.
BACKOFF → "How long do I WAIT between retries?"
naive : 0ms, 0ms, 0ms ... (hammer immediately)
backoff: 250ms, 500ms, 1s, 2s (wait longer each time)
JITTER → "Do all clients wait the EXACT same amount?"
no jitter: every client retries at t=250ms → they sync up
jitter : each waits 250ms ± a random wobble → they spread out
Put simply with a teaching analogy: imagine a door that jams. Timeout is how long you push before letting go. Retry is deciding to push again. Backoff is waiting a bit longer before each push instead of slamming it nonstop. Jitter is making sure you and everyone else in the crowd don't all shove at the exact same second — because that is what breaks the door.
⚠️ Backoff without jitter is only half a fix. If every client backs off by exactly 250ms, then 500ms, then 1s, they stay perfectly synchronized — the herd just stampedes in waves. Jitter is the small random offset that breaks the lockstep so the load smears out instead of arriving in spikes.
🧪 Verification Discipline — same rule, new session
As always, we do not prove this with throwaway PowerShell. Session 6 gets its own repeatable scenarios under tools/Wassal.SessionTests, sharing the same engine and reporting as Session 5. Each scenario cleans the DB, fires the requests in a repeatable way, prints a readable console table, applies clear pass/fail rules, and saves a timestamped report under Reports/Session-06/ that previous runs never overwrite — so the evidence is easy to attach to this writeup.
One engine, two scenarios — different verdicts. The shared mechanics (clean → simulate N clients × A attempts reusing one key per client → record when each attempt was sent → count rows) live once in RetryStormCore. Each scenario reuses that engine but brings its own verdict. The only behavioural difference between the two is the retry timing policy — naive (no wait) vs resilient (backoff + jitter). No server change is needed between them.
⚖️ Lesson 6 has TWO scenarios with different verdict logic
Both simulate the same clients making the same number of attempts with the same reused key. What differs is what counts as success:
① BEFORE — Naive retry storm
dotnet run --project tools/Wassal.SessionTests -- session-06-naive-retry-storm
A diagnostic. Its job is to expose concentrated load, so it is happy when the attempts pile up. PASS when:
- Attempts are crammed together (avg spacing < 50 ms)
- Many attempts land in one short window (load concentration)
- Duplicates still avoided (DB rows = number of clients)
Meaning: idempotency kept it correct, but the retries arrived as one spike. Correctness ≠ scalability.
② AFTER — Resilient retry
dotnet run --project tools/Wassal.SessionTests -- session-06-resilient-retry
Short alias: session-06. A proof. It is happy only when correctness and spread both hold. PASS when:
- DB row count == clients, unique ids == clients
- Every client eventually succeeded
- Retries spread out (avg spacing ≥ 50 ms)
⚠️ Same input, opposite verdicts — on purpose. The 50 ms boundary is the hinge: the naive run sits below it (a spike), the resilient run sits above it (spread out). Both keep the database correct — that is the whole lesson. The improvement you are proving is about load, not data.
Task 1 — Feel the Naive Retry Storm (BEFORE) (10 min)
Make sure the Wassal API is running (the same idempotent server from Lesson 5 — no changes needed):
dotnet run --project src/Wassal.Monolith --launch-profile https
Now run the naive scenario. It simulates 5 clients, each retrying its order 5 times, with no wait between attempts:
# From the repo root
dotnet run --project tools/Wassal.SessionTests -- session-06-naive-retry-storm
Optional overrides (defaults come from appsettings.json):
dotnet run --project tools/Wassal.SessionTests -- session-06-naive-retry-storm `
--clients 5 `
--attempts 5 `
--cleanDb true
The console prints a per-client summary, an attempt timeline sorted by send time (so you can literally see them stack up), and a metrics block. It then saves a report like Reports/Session-06/Session-06-Naive-Retry-Storm-RunReport-2026-06-21-143012.txt.
🐛 The storm. You'll see all 25 attempts land almost on top of each other — a tiny
avg spacing(single-digit ms) and a highMax attempts in 200ms window. The database is still correct (5 orders, one per client, duplicates avoided) because each client reused its key. But the load was concentrated into one spike. On a server that is busy or just recovering from a blip, that spike is the thundering herd that turns a hiccup into a full outage.💡 Read the verdict carefully. This scenario passes (exit 0) when the storm IS observed — that's the point of a BEFORE diagnostic. "Attempts too close together + duplicates still avoided" is the documented bad state we want to make visible before we fix it.
✅ Done when: you've seen the attempts cram into one window, a tiny avg spacing, and DB rows = number of clients (duplicates avoided).
Task 2 — Write a Hand-Made Backoff + Jitter Loop (15 min)
The fix is entirely on the client. We change one thing: how long a client waits before each retry. We keep it hand-written on purpose — no Polly, no circuit breaker — so the mechanics stay visible. Here is the whole idea in a few lines:
// the resilient retry policy (illustrative)
// One key for the whole operation — generated ONCE, reused on every retry.
var idempotencyKey = Guid.NewGuid().ToString();
const int baseDelayMs = 250; // first backoff gap
const double factor = 2.0; // double the wait each time (exponential)
const int maxAttempts = 5; // bounded — never retry forever
var rng = new Random();
for (var attempt = 1; attempt <= maxAttempts; attempt++)
{
// 1) WAIT before retrying (the first attempt waits 0).
if (attempt > 1)
{
var gap = baseDelayMs * Math.Pow(factor, attempt - 2); // 250, 500, 1000, 2000
var jitter = 0.6 + rng.NextDouble() * 0.8; // random ×[0.6 .. 1.4)
await Task.Delay((int)(gap * jitter)); // backoff + jitter
}
// 2) Send the SAME request with the SAME key.
var result = await PostOrderAsync(body, idempotencyKey);
// 3) Stop as soon as it works; otherwise loop and back off again.
if (result.Ok) return result;
}
// Bounded: after maxAttempts we give up gracefully (surface a clean error,
// not an infinite stampede).
Three things make this "resilient" instead of "naive":
-
Bounded —
maxAttemptsmeans we never retry forever. - Exponential backoff — each gap is roughly double the last (250ms → 500ms → 1s → 2s), so a struggling server gets breathing room that grows.
-
Jitter — the
±40%random multiplier means clients don't retry in lockstep, so the load smears out instead of arriving in synchronized waves.
💡 In this repo the exact same policy already lives in the scenario class
Session06ResilientRetryScenario(methodResilientDelay), and the naive one inSession06NaiveRetryStormScenario(NaiveDelayreturns0). The shared firing engine isRetryStormCore. You don't have to edit the server at all — the difference between BEFORE and AFTER is purely this delay function.
✅ Done when: you can explain, in one sentence each, what bounded / backoff / jitter add — and why the key is created once, not per attempt.
Task 3 — Prove Resilient Retry (AFTER) (10 min)
Run the resilient scenario — same clients, same attempts, same per-client key, only the timing policy changed:
# SAME runner, SAME engine — different policy, different verdict
dotnet run --project tools/Wassal.SessionTests -- session-06-resilient-retry
# Short alias:
dotnet run --project tools/Wassal.SessionTests -- session-06
This scenario applies 5 explicit pass/fail checks:
-
DB row count == clients— one order per client -
Unique ids == clients— every client's retries point at its one order -
Duplicates avoided— idempotency held under retries -
All clients succeeded— every operation eventually got a result -
Retries spread out— average spacing ≥ 50 ms (backoff actually applied)
✅ Both halves hold. Read it against the timeline: the same 25 attempts are now smeared across a few seconds instead of one burst.
avg spacingjumps from single-digit ms to well over the 50 ms boundary, andMax attempts in 200ms windowdrops sharply. Meanwhile the DB still shows one order per client. Idempotency kept it correct; backoff + jitter kept it gentle. Exit code 0 means all five checks passed.⚠️ If you ran the naive scenario after this, it would "fail" its storm check — because there's no storm left to find. That failure is good news. Always match the scenario to what you're testing: naive to expose the spike, resilient to prove the spread.
✅ Done when: all 5 checks PASS — duplicates avoided AND retries spread out (avg spacing ≥ 50 ms).
Task 4 — Why the Same Idempotency-Key Must Survive Every Retry (10 min)
This is the single most important rule of client retries, and the easiest to get wrong. The Idempotency-Key belongs to the operation, not to the attempt. The client must generate it once, before the first try, and reuse the exact same value on every retry of that operation.
✅ CORRECT — key created once, reused on every retry
operation: "create my order" → key = ABC-123 (generated ONE time)
attempt 1 → POST Idempotency-Key: ABC-123
attempt 2 → POST Idempotency-Key: ABC-123 ← same key
attempt 3 → POST Idempotency-Key: ABC-123 ← same key
Server: "I've seen ABC-123 → replay the existing order." → 1 order ✅
Because the key is stable, the server recognizes attempts 2 and 3 as the same intent and replays the first result. That is exactly the replayed=true behaviour from Lessons 4 and 5 — and it's why our resilient scenario produces one order per client no matter how many times each client retries.
⚠️ The anti-pattern: a new key per retry. If the client generates a fresh key for each attempt, every retry looks like a brand-new operation to the server — and idempotency can't help, because dedup is keyed on… the key.
❌ WRONG — a new key every attempt
operation: "create my order"
attempt 1 → POST Idempotency-Key: KEY-1 → Server: new! → Order #55
attempt 2 → POST Idempotency-Key: KEY-2 → Server: new! → Order #56
attempt 3 → POST Idempotency-Key: KEY-3 → Server: new! → Order #57
Result: 1 user intent → 3 DUPLICATE orders (and 3 charges). 💥
Here the customer wanted one Koshary + Pizza order and got billed three times. The server did nothing wrong — it was told, three times, "this is a new order." Generating a new key per retry doesn't just weaken idempotency; it completely defeats it. All the work from Lessons 3–5 is undone by one client-side mistake.
💡 Rule of thumb. New key = new intent. Same retry = same key. If you can't point to where the key was generated once and held across retries, your "idempotent" client isn't. In our runner, that's exactly why each simulated client builds one key (
…-client-N) and reuses it for all of its attempts.
✅ Done when: you can state why reusing the key gives 1 order, and why a new key per retry gives N duplicates.
Task 5 — Capture the Evidence, Tag, Document (5 min)
Unlike Lesson 5, there is no server change today — the BEFORE and AFTER are two client policies run against the same idempotent API. So the "before/after" lives in the two saved reports under Reports/Session-06/, not in a controller diff. Attach both reports to the writeup as evidence.
Commit the new scenarios + session plan, and tag the AFTER state:
cd D:\books\distributed-system\Wassal
git add tools/Wassal.SessionTests/ docs/sessions/Day-06-Timeouts-Retries-Backoff-Jitter.html
git commit -m "feat: Lesson 6 — client-side resilient retries (backoff + jitter) + scenarios"
git tag lesson-06-after
Suggested lesson writeup — docs/lessons/lesson-06-timeouts-retries-backoff-jitter.md:
# Lesson 6 — Timeouts, Retries, Backoff & Jitter
**Tags:** `lesson-06-before` (naive) → `lesson-06-after` (resilient)
**Builds on:** Lesson 5 (`lesson-05-after`)
## The pain (BEFORE)
Naive retries (no wait) from several clients pile every attempt into one
tiny window. Idempotency still keeps the DB correct — one order per client,
duplicates avoided — but the LOAD is concentrated into a spike: the
thundering herd. Correct is not the same as scalable.
## The fix (AFTER)
A bounded, hand-written retry loop with exponential backoff (250ms × 2 each
time) and ±40% jitter, reusing ONE Idempotency-Key per operation. Same
attempts, same correctness, but the load is spread over time instead of
stacked. Verified by `session-06-resilient-retry`: DB rows = clients,
unique ids = clients, every client succeeded, avg spacing ≥ 50ms.
| Scenario | Correctness | Load |
|--------------------------------|--------------------|------------------------------|
| Naive retry (no wait) | 1 order/client ✅ | concentrated spike ❌ |
| Resilient (backoff + jitter) | 1 order/client ✅ | spread over time ✅ |
## The key rule
Generate the Idempotency-Key ONCE per operation and reuse it on every
retry. A new key per retry turns 1 intent into N duplicate orders and
defeats everything from Lessons 3–5.
## What's still deliberately missing (later lessons)
- **Polly / resilience libraries.** We hand-rolled the loop to see it.
Polly comes later (Part 6, Circuit Breakers).
- **Circuit breaker.** Stop calling a service that is clearly down. Later.
- **Server-side rate limiting / 429.** Defending the server FROM abusive
clients is a separate lesson (Part 6).
- **Key TTL / expiry.** Still parked for the caching lesson.
✅ Done when: both Session-06 reports are saved, the scenarios + session plan are committed, and lesson-06-after is tagged.
Quick Checklist
| # | Task | Time | Done When |
|---|---|---|---|
| 1 | Run naive storm → expose concentrated load | 10 min | Tiny avg spacing + DB rows = clients (duplicates avoided) |
| 2 | Understand + write the backoff + jitter loop | 15 min | You can explain bounded / backoff / jitter |
| 3 | Run resilient retry → prove the fix | 10 min | All 5 checks PASS (correct AND spread out) |
| 4 | Why reuse the key; why a new key per retry breaks it | 10 min | Can explain 1 order vs N duplicates |
| 5 | Save reports, commit, tag lesson-06-after
|
5 min | Both reports saved + writeup committed |
🎓 The compounding effect — 6 lessons in. Your idempotent POST endpoint now survives app restarts (Lesson 4), body tampering (Lesson 4), and genuine concurrency (Lesson 5) — and you finally have the client that knows how to retry into it safely (Lesson 6). The server guarantees correctness; the client guarantees politeness. That two-sided contract — at-least-once delivery + idempotent handling + backoff — is the real shape of "exactly-once effect" that production systems actually ship.
Part of the **Fundamentals of Distributed Systems* series — building Wassal, a distributed food-delivery lab, one concept at a time.*
Top comments (0)