DEV Community

mohamed Tayel
mohamed Tayel

Posted on

Day 7 — Part 2: The Polite Client Obeys Retry-After

#dotnet #distributedsystems #resilience #tutorial

The server already learned to say "not now." Now the client learns to listen.

Retry-After — the polite client waits

⚠️ This is Day 7 — Part 2, not Lesson 8. It is a direct continuation of the 429 + Retry-After lesson from Part 1. Part 1 was the server side (rate limiting); Part 2 is the client side (cooperating with the server's signal). Same lesson, second half.

Recap from Day 7 · Part 1

In Part 1 we taught the server to defend itself with a token-bucket rate limiter on POST /api/orders:

  • When the bucket runs dry, the server returns HTTP 429 Too Many Requests with a Retry-After header.
  • Rejected requests did no database work — a 429 creates no order, so there is never a half-finished mess to clean up.
  • The server stayed healthy under a flood by shedding the excess instead of trying to serve everything.

The Part 1 numbers, side by side:

Run Requests Accepted (200) Rejected (429) Retry-After DB rows
BEFORE (no limiter) 30 30 0 — 30
AFTER (token bucket) 30 5 25 25 / 25 5

That AFTER row is exactly where Part 2 begins. The server protected itself — but 25 requests walked away with a 429. What happens to those 25? That depends entirely on the client.

💡 A note on the numbers. For the Part 2 demo we use a smaller, faster run — 12 logical operations instead of 30 — so the retry behaviour is easier to read in the console. The lesson is the same: some requests hit 429; the naive client loses them, while the polite client waits and completes them.

The one idea to take away

💡 429 is not "never." 429 is "not now." The server already did its job in Part 1. In Part 2 the client does its job: read Retry-After, wait, and retry the same operation with the same key — until it completes.

Skills you'll lean on again (Lessons 3–7)

  • Idempotency-Key = one key per business operation
  • Idempotency makes a retry safe (no duplicates)
  • Backoff + jitter to avoid a stampede (Lesson 6)
  • 429 + Retry-After from the server (Part 1)
  • Before/After discipline + saved reports

Skills you'll add in Part 2

  • Reading the Retry-After header on the client
  • Waiting at least that long before retrying
  • Retrying the SAME operation with the SAME key
  • Bounding attempts so the client never loops forever
  • Turning a 429 into an eventual success

Part 2 Goal

Take the Part-1 server exactly as it is (no server change) and put two clients in front of it. First a naive client that treats a 429 as a final failure — and loses operations. Then a polite client that reads Retry-After, waits, and retries the same operation with the same Idempotency-Key — until every operation completes, with no duplicates.

The Problem — a naive client throws work away

A naive client sees a non-200 status and gives up. To it, 429 looks like "failed," so it reports failure to the user and stops. But the server didn't say the operation was impossible — it said "not right now."

12 logical operations, fired at a rate-limited server.

bucket allows ~5 now → 5 operations get 200 OK  → completed
the other ~7 hit the empty bucket → 429          → naive client GIVES UP

Result with a naive client:
  completed = 5
  lost      = 7   ← thrown away, never retried
  DB rows   = 5   ← only the accepted operations created orders
Enter fullscreen mode Exit fullscreen mode

🐛 The loss is silent. The data that does exist is correct (5 orders for 5 accepted requests), so nothing looks broken on the server. But 7 real customer intents just vanished because the client mistook "not now" for "never." DB rows match only the accepted requests — not what the user actually asked for.

Mental Model — 429 means "not now," not "never" 

403 Forbidden        → "No. You are not allowed." (never)
404 Not Found        → "That doesn't exist."       (never, here)
429 Too Many Requests→ "Not now — try again soon." (later!)

Retry-After: 1       → "Here is exactly how long to wait: ~1 second."
Enter fullscreen mode Exit fullscreen mode

This is the key reframe. A 429 is a temporary answer with an expiry date attached. The server is not rejecting the operation forever — it is asking the client to come back. And it even tells the client when via Retry-After. A good client treats that as an instruction, not an error.

السيرفر مش بيقولك مستحيل. هو بيقولك: استنى شوية وجرب تاني.

(The server isn't telling you "impossible." It's telling you: wait a little and try again.)

The Correct Client Behavior

For each logical operation, the polite client follows this loop:

  1. Generate one Idempotency-Key for the operation (once, up front).
  2. Send the request.
  3. If 200 OK → done.
  4. If 429 →
    • read Retry-After
    • wait Retry-After seconds
    • optionally add a small jitter on top (so clients don't all return on the same tick)
    • retry using the SAME Idempotency-Key
  5. Stop after a max attempt count (never loop forever).

In pseudocode (illustrative — the real loop lives in the verification scenario, not the app):

// One key for the whole operation — generated ONCE, reused on every retry.
var idempotencyKey = Guid.NewGuid().ToString();
const int maxAttempts = 5;

for (var attempt = 1; attempt <= maxAttempts; attempt++)
{
    var response = await PostOrderAsync(body, idempotencyKey);

    if (response.Status == 200) return response;        // ✅ completed

    if (response.Status == 429)
    {
        var retryAfter = response.RetryAfterSeconds ?? 1; // server told us when
        var jitter     = Random.NextDouble() * 0.25;      // small wobble (Lesson 6)
        await Task.Delay(TimeSpan.FromSeconds(retryAfter + jitter));
        continue;                                         // try the SAME key again
    }

    throw new Exception($"Unexpected status {response.Status}");
}
// Bounded: after maxAttempts, give up gracefully (don't hammer forever).
Enter fullscreen mode Exit fullscreen mode

⚠️ Wait at least Retry-After — and add jitter on top. Waiting less than Retry-After disobeys the server and just earns another 429. Waiting the exact same value on every client re-creates the thundering herd from Lesson 6 — they all come back on the same tick. So: honor Retry-After as the floor, then add a little randomness.

Why the SAME Idempotency-Key matters

The Idempotency-Key belongs to the operation, not the attempt. Generate it once, reuse it on every retry of that operation.

✅ CORRECT — same key across retries

operation "create my order"  →  key = ABC-123  (generated once)
   attempt 1 → 429 (not now)
   attempt 2 → POST  Idempotency-Key: ABC-123   ← same key
   attempt 3 → 200 OK → Order #55
Result: exactly 1 order. ✅
Enter fullscreen mode Exit fullscreen mode 
❌ WRONG — a new key per retry

operation "create my order"
   attempt 1 → key KEY-1 → 429
   attempt 2 → key KEY-2 → 200 → Order #55
   attempt 3 → key KEY-3 → 200 → Order #56
Result: the server sees 3 DIFFERENT operations → duplicate orders. 💥
Enter fullscreen mode Exit fullscreen mode

Because the key is stable, the server recognizes a retry as the same intent and never creates a second order. This is the guarantee built in Lessons 3–5 (in-memory dedup → durable store → concurrency-safe replay). Part 2 simply depends on it: idempotency is what makes "just retry" a safe thing to do.

💡 Rule of thumb. New key = new intent. Same retry = same key. If you can't point to where the key was created once and held across retries, your "safe retry" isn't safe.

Verification Discipline — same rule, Part 2

As always, we prove this with repeatable .NET scenarios, not throwaway PowerShell. Part 2 adds two scenarios that run against the unchanged Part-1 rate-limited server — only the client behaviour differs. Each cleans the DB, fires the operations, prints a readable table, applies a clear verdict, and saves a timestamped report that previous runs never overwrite.

The naive scenario expects loss (operations thrown away). The polite scenario expects full completion (every operation eventually succeeds, no duplicates). The only thing that changes is whether the client obeys Retry-After.

Two scenarios with different verdict logic

Both fire the same 12 operations at the same rate-limited server. What differs is how the client reacts to a 429:

① BEFORE — Naive gives up (a diagnostic; happy when loss is demonstrated)

dotnet run --project tools/Wassal.SessionTests -- session-07-part-2-naive-gives-up
Enter fullscreen mode Exit fullscreen mode

Verdict — PASS when:

  • At least one 429 observed
  • completed > 0 and lost > 0
  • DB rows == completed (lost ops created nothing)

② AFTER — Polite obeys Retry-After (a proof; happy only when everything completes, exactly once)

dotnet run --project tools/Wassal.SessionTests -- session-07-part-2-polite-retry-after

# Short alias:
dotnet run --project tools/Wassal.SessionTests -- session-07-part-2
Enter fullscreen mode Exit fullscreen mode

Verdict — PASS when:

  • completed == total operations
  • DB rows == total, unique ids == total
  • every retry waited ≥ Retry-After; attempts > operations

⚠️ Same input, opposite verdicts — on purpose. Both keep the database correct (no duplicates). The naive client just leaves work unfinished; the polite client finishes it. The improvement Part 2 proves is completion, not correctness.

BEFORE — Watch the Naive Client Lose Operations

Start the Part-1 server (token bucket already in place — no change):

dotnet run --project src/Wassal.Monolith --launch-profile https
Enter fullscreen mode Exit fullscreen mode

Then run the naive scenario:

dotnet run --project tools/Wassal.SessionTests -- session-07-part-2-naive-gives-up
Enter fullscreen mode Exit fullscreen mode

What it does: fires 12 logical operations (each its own Idempotency-Key), one attempt each. A 200 is completed; a 429 is lost-429 (no retry).

Expected shape:

  • 12 logical operations
  • some complete (≈5), some get 429 (≈7)
  • lost > 0
  • DB rows == completed only

🐛 Diagnostic PASS = loss observed. Like every BEFORE scenario, this one "passes" when it proves the problem: at least one 429, some completed, some lost, and the DB holding only the completed operations. That is the naive client throwing work away.

✅ Done when: lost > 0 and DB rows == completed (the lost operations created nothing).

AFTER — The Polite Client Completes Everything

Same server, same 12 operations — now with a client that listens:

dotnet run --project tools/Wassal.SessionTests -- session-07-part-2-polite-retry-after

# Short alias:
dotnet run --project tools/Wassal.SessionTests -- session-07-part-2
Enter fullscreen mode Exit fullscreen mode

What it does: for each operation, on a 429 the client reads Retry-After, waits that long (plus a little jitter), and retries with the same Idempotency-Key — up to a bounded max attempts.

Expected shape:

  • 12 logical operations
  • 429s may happen as intermediate attempts (not final)
  • the client waits according to Retry-After
  • the same Idempotency-Key is reused per operation
  • all operations eventually complete
  • DB rows == total operations
  • unique order ids == total operations
  • attempts > operations (retries actually happened)

✅ Loss → zero. The same flood that left 7 operations behind in the naive run now completes all 12. Idempotency keeps it to exactly 12 orders (no duplicates), and obeying Retry-After keeps the client from hammering the recovering server. The 429s become speed bumps, not dead ends.

✅ Done when: completed == 12, DB rows == 12, unique ids == 12, and attempts > 12.

Console & Report Expectations

Each scenario prints a per-operation table (operation number, short key, status, result, Retry-After seen, attempts, elapsed ms, order id) plus a summary block and a verdict. Reports save automatically — timestamped, never overwritten — under:

Reports/Session-07-Part-2/
Enter fullscreen mode Exit fullscreen mode

Example file names:

Session-07-Part-2-Naive-Gives-Up-RunReport-yyyy-MM-dd-HHmmss.txt
Session-07-Part-2-Polite-Retry-After-RunReport-yyyy-MM-dd-HHmmss.txt
Enter fullscreen mode Exit fullscreen mode

💡 Why save reports? Same discipline as every session: a saved, timestamped report is evidence you can attach to this writeup and compare run-to-run. The naive report shows loss; the polite report shows full completion — side by side they tell the whole Part-2 story.

Implementation Walkthrough — What We Built & Tested

A beginner-friendly tour of the two verification scenarios behind Day 7 Part 2 — the naive client that loses work, and the polite client that obeys Retry-After and completes everything exactly once. The server (the Part 1 token-bucket rate limiter) was not changed — Part 2 is about client behaviour only.

The whole idea in one line: Part 1 taught the server to say "not now" with HTTP 429 + Retry-After. Part 2 is about the client: does it listen? We built two clients and ran them against the same rate-limited server to find out.

  • Naive client — treats 429 as a final failure and throws the work away.
  • Polite client — reads Retry-After, waits, adds a little jitter, and retries the same operation with the same Idempotency-Key until it completes.

BEFORE — The Naive Client Gives Up

File: tools/Wassal.SessionTests/Sessions/Session07Part2NaiveGivesUpScenario.cs

Command:

dotnet run --project tools\Wassal.SessionTests -- session-07-part-2-naive-gives-up
Enter fullscreen mode Exit fullscreen mode

What it does:

  • Sends 12 logical operations concurrently at the rate-limited POST /api/orders.
  • Each operation has its own Idempotency-Key (distinct operations, not retries of one).
  • Each operation makes only one attempt.
  • If the server returns 200 → the operation is completed.
  • If the server returns 429 → the operation is marked lost-429.
  • No retry happens. The naive client gives up.

🐛 Why this "passes" by failing. This is a BEFORE diagnostic. It is designed to prove the problem exists. PASS here means the bug was demonstrated: the naive client mistook 429 ("not now") for a permanent failure and lost real customer work.

Verified result:

Metric Value Meaning
Total operations 12 Everything we fired
Completed (200) 5 Got through the bucket
Lost (429, gave up) 7 Thrown away — never retried
DB rows (Orders) 5 == completed only
Unique order ids 5 No duplicates, but work is missing

VERDICT: NAIVE LOSS DEMONSTRATED ✅ (diagnostic)

The data that exists is correct — DB rows == completed — so nothing looks broken on the server. But 7 real intents silently vanished. That silent loss is exactly what the polite client fixes. (The 5/7 split can vary slightly with timing; the verdict only requires some loss, not exactly 5 and 7.)

AFTER — The Polite Client Obeys Retry-After

File: tools/Wassal.SessionTests/Sessions/Session07Part2PoliteRetryAfterScenario.cs

Command:

dotnet run --project tools\Wassal.SessionTests -- session-07-part-2-polite-retry-after
Enter fullscreen mode Exit fullscreen mode

Short alias (runs the same polite scenario):

dotnet run --project tools\Wassal.SessionTests -- session-07-part-2
Enter fullscreen mode Exit fullscreen mode

What it does:

  • Sends the same 12 logical operations concurrently at the same Part 1 server.
  • Each operation gets one Idempotency-Key generated once, up front.
  • If the server returns 200 → the operation is completed.
  • If the server returns 429, the client:
    • reads Retry-After,
    • waits at least Retry-After seconds,
    • adds a small jitter on top,
    • retries using the SAME Idempotency-Key.

The knobs (explicit and visible in code):

  • Bounded retries: MaxAttempts = 6 — the loop can never run forever.
  • Jitter: 0..250 ms added on top of Retry-After (never less than it).
  • Fallback: if Retry-After is missing or unparsable → use 1 second.
  • Never generate a new key per retry — a retry is the same intent.

Verified result:

Metric Value Meaning
Total operations 12 Same as the naive run
Completed (200) 12 Every operation finished
429 intermediate attempts 9 Speed bumps, not dead ends
Total attempts (POSTs) 21 Retries actually happened
DB rows (Orders) 12 == total operations
Unique order ids 12 No duplicates
Every retry honored Retry-After YES The client obeyed the server

VERDICT: PASS ✅

✅ 21 attempts → only 12 orders. The client made 21 HTTP attempts, but the database holds only 12 orders with 12 unique ids. That gap is the proof of correctness: every retry reused the same Idempotency-Key, so the server recognized the retry as the same operation and never created a duplicate. Completion went up; duplicates stayed at zero.

💡 Same flood, opposite outcome. The naive run left 7 operations behind. The polite run — same 12 operations, same rate limiter — completed all 12 by turning each 429 into a short wait and a retry.

How to Run It Locally

You need two terminal windows: one keeps the server running, the other runs the scenarios.

Window 1 — start the server (keep this open):

cd /d D:\books\distributed-system\Wassal\src
dotnet run --project Wassal.Monolith --launch-profile https
Enter fullscreen mode Exit fullscreen mode

Wait until you see this line — it means the server is ready:

Now listening on: https://localhost:7126
Enter fullscreen mode Exit fullscreen mode

⚠️ Leave Window 1 open. The server must stay running the whole time you run the scenarios in Window 2. If you close it (or press Ctrl+C), every scenario will fail to connect to https://localhost:7126.

Window 2 — build, then run the scenarios:

cd /d D:\books\distributed-system\Wassal

dotnet build

REM BEFORE — naive client (should show loss)
dotnet run --project tools\Wassal.SessionTests -- session-07-part-2-naive-gives-up

REM AFTER — polite client (should complete everything)
dotnet run --project tools\Wassal.SessionTests -- session-07-part-2-polite-retry-after

REM AFTER via the short alias (same polite scenario)
dotnet run --project tools\Wassal.SessionTests -- session-07-part-2

REM Part 1 regression — the server should still shed load correctly
dotnet run --project tools\Wassal.SessionTests -- session-07-rate-limited
Enter fullscreen mode Exit fullscreen mode

💡 What to expect. The naive run prints NAIVE LOSS DEMONSTRATED ✅, the polite run and its alias print PASS ✅, and the Part 1 regression still prints PASS ✅. Each run also saves a timestamped report.

What I Should Understand Before Moving On

  • 429 means "not now", not "never" — a temporary answer with an expiry attached; come back soon, the door isn't locked forever.
  • Retry-After tells you how long to wait — the server hands the client an exact instruction. Treat it as a rule, not noise.
  • Naive client ignores Retry-After — it treats 429 as final and loses real work; completion drops, silently.
  • Polite client obeys Retry-After — it waits, retries, and completes every operation. 429 becomes a speed bump.
  • The key belongs to the operation — Idempotency-Key identifies the logical operation, not a single HTTP attempt.
  • Same retry = same key · New key = new intent — reuse the key on retries so the server sees one operation. A new key means a new order.
  • Bounded retries prevent infinite loops — MaxAttempts = 6 guarantees the client eventually stops instead of hammering forever.
  • Jitter de-synchronizes clients — a small random add-on stops every client from retrying on the exact same tick (no new stampede).
  • DB rows are the proof — naive — DB rows = completed only. The lost operations created nothing.
  • DB rows are the proof — polite — DB rows = total operations, with unique ids = total operations.

الخلاصة بالمصري 🇪🇬

السيرفر لما يرجع 429 هو مش بيقول إن العملية فشلت للأبد. هو بيقول للعميل: استنى شوية وجرب تاني. العميل الغلط بيسيب الطلب يضيع. العميل الصح بيستنى Retry-After ويرجع يحاول بنفس Idempotency-Key، علشان السيرفر يفهم إن دي نفس العملية مش طلب جديد. عشان كده كل العمليات كملت، والداتابيز فيها 12 أوردر فقط من غير أي تكرار.

Final Comparison — Naive vs Polite

Scenario Behavior on 429 Completed Lost Attempts DB Rows Verdict
Naive Gives up 5 7 12 5 NAIVE LOSS DEMONSTRATED ✅
Polite Waits + retries same key 12 0 21 12 PASS ✅

Read the two rows together: the polite client made more attempts (21 vs 12) but produced the same kind of clean database — no duplicates — while finishing every operation (12 vs 5). More attempts, zero duplicates, full completion. That is the whole lesson in one table.

Final takeaway — the whole contract

💡 Backoff is the client choosing to slow down (Lesson 6). Rate limiting is the server enforcing the slowdown (Part 1). 429 + Retry-After is the contract between them.

The polite client completes the operation safely by listening to the server — waiting Retry-After — and reusing the same Idempotency-Key so the eventual retry creates exactly one order. Server protection (Part 1) + client cooperation (Part 2) = every operation completes, exactly once, even through a rate limiter.

Part 1 gave the server a voice (429 + Retry-After). Part 2 gave that voice a listener. Together they complete the rate-limiting lesson. This is not Lesson 8 — it's the second half of Day 7. Lesson 8 will move on to the next topic.

Part of the **Fundamentals of Distributed Systems* series — building Wassal, a distributed food-delivery lab, one concept at a time.*

Top comments (0)