Day 7 — When Politeness Isn't Enough: Rate Limiting & Load Shedding

Lesson 6 taught the client to back off — but you can't trust every client to. Today the server learns to defend itself: shed the excess, and tell callers exactly when to come back.

Bots, buggy SDKs, and panicking apps will still hammer you. In this session we flood the order endpoint and watch the unprotected server accept every single request — no backpressure at all. Then we add a token-bucket rate limiter using .NET 8's built-in middleware, so the excess gets 429 + Retry-After and the load is shed. This is Part 1 (the server side); Part 2 is the client side.

📘 Where we left off in Lesson 6: You built the client's half of idempotency: a bounded retry loop with exponential backoff + jitter, reusing one Idempotency-Key per operation. Naive retries piled into one window (a storm); polite retries spread out over time. Tagged as lesson-06-after. But Lesson 6 ended on an uncomfortable admission: "you can't trust every client to back off." A polite client cooperates — a bot doesn't. So the server cannot rely on good manners. It needs its own defence, one it enforces no matter who is calling. That defence is rate limiting, and the way it cooperates with polite clients is 429 + Retry-After.

🎯 The one idea to take away: Backoff is the client choosing to slow down. Rate limiting is the server enforcing it. You cannot outsource load control to clients. The server must have backpressure of its own — and 429 + Retry-After is how the two sides cooperate. Same theme as Lesson 6, now from the other side of the wire: idempotency keeps it correct; rate limiting keeps it standing.

---

What you'll reinforce and add

🔁 Reinforce — from Lessons 5 & 6:

• Idempotency keeps accepted work correct (no duplicates)
• Bursts of concurrent requests from the .NET runner
• DB row count as the authoritative check
• Before/After discipline + saved reports
• Reading a console table + verdict

✨ New — added today:

• Backpressure & why client politeness can't be trusted
• The token bucket algorithm (rate + burst)
• .NET 8's built-in rate limiter middleware
• 429 Too Many Requests + Retry-After
• Load shedding: protect the majority by failing the excess fast

🎯 Today's goal: Flood the order endpoint with a burst of requests and watch the unprotected server accept every single one — no backpressure at all. Then add a token-bucket rate limiter using .NET 8's built-in middleware, so the excess gets 429 + Retry-After and the load is shed. Prove that the accepted requests still each create exactly one order (rejected requests do no work) — and see how the Retry-After header is exactly the signal a Lesson-6 client backs off on.

---

🧠 Mental Model 1 — Why client politeness isn't enough

Lesson 6 made our client polite. But your server doesn't only talk to your client. It talks to whatever shows up:

Good mobile app   →  retries with backoff + jitter   (polite, Lesson 6)
Old cached build  →  retries instantly, no backoff    (storm)
Third-party SDK   →  retries 50× in a tight loop       (worse storm)
A bot / scraper   →  10,000 requests/sec, doesn't care (attack)

The server cannot CHOOSE its callers. Politeness is voluntary.
So load control cannot live only on the client — the server needs
its OWN limit that it enforces on everyone, no exceptions.

That server-side limit is called backpressure: the system's ability to say "I am full, slow down" instead of trying to accept everything and collapsing. Without backpressure, one rude caller can consume all the capacity and starve every well-behaved user. Idempotency won't save you here — it keeps the data correct, but it happily does the (correct) work for all 10,000 requests. Correct, and on fire.

💡 Fail fast beats slow death. When overloaded, a server that rejects the excess in microseconds stays healthy for everyone it does serve. A server that tries to serve everyone slows to a crawl and fails for everyone. Rejecting some requests on purpose to protect the rest is called load shedding — and it is a feature, not a failure.

---

🧠 Mental Model 2 — The token bucket & the 429 + Retry-After contract

The classic way to enforce "X requests per second, with a little burst room" is a token bucket. Picture a bucket that holds a few tokens. Every request must take one token to be served. The bucket refills at a steady rate:

Bucket capacity = 5 tokens      (max burst it will absorb at once)
Refill          = 5 tokens / second  (steady allowed rate)

request arrives → is there a token?
   YES → take one, serve it (200 OK)
   NO  → bucket empty → REJECT with 429 Too Many Requests
                        + Retry-After: 1   ("come back in ~1s")

A burst of 30 at once → ~5 served now, ~25 rejected with 429.
The bucket refills, so a polite client that waits gets served next round.

The teaching analogy: the bucket is a doorman holding 5 entry tickets. Five people walk straight in; the rest are told "no tickets right now — come back in a second" (that's the Retry-After). The doorman prints 5 new tickets every second. Nobody is hurt, and the room never overflows.

⚠️ 429 + Retry-After is a contract, not just an error. A bare rejection says "go away." 429 + Retry-After: 1 says "go away for one second, then you're welcome." That second value is precisely what a Lesson-6 client feeds into its backoff: instead of guessing how long to wait, it obeys the number the server gave it. Server and client cooperate — one sets the pace, the other respects it.

---

🧪 Verification Discipline — same rule, new session

As always: no throwaway PowerShell. Session 7 gets its own repeatable scenarios under tools/Wassal.SessionTests, sharing the same engine and reporting as Sessions 5 & 6. Each scenario cleans the DB, fires the burst in a repeatable way, prints a readable table (including the Retry-After column), applies clear pass/fail rules, and saves a timestamped report under Reports/Session-07/ that previous runs never overwrite.

One engine, two scenarios — different verdicts. The shared mechanics (clean → fire one burst of distinct requests → record 2xx / 429 + Retry-After → count rows) live once in FloodCore. Each scenario reuses that engine but brings its own verdict. The only difference between BEFORE and AFTER is whether the rate limiter is switched on in the server — which is a reader step below, exactly like the Day-05 controller edit.

⚖️ Lesson 7 has TWO scenarios with different verdict logic

Both fire the same burst of distinct requests. What differs is what counts as success:

① BEFORE — Unprotected flood

dotnet run --project tools/Wassal.SessionTests -- session-07-unprotected-flood

A diagnostic. Its job is to show there's no defence, so it's happy when everything gets through. PASS when:

• Nothing was shed (0 × 429)
• Every request accepted (accepted == total)
• DB rows == accepted (each made one order)

Meaning: the server has no backpressure. A bot could soak all capacity.

② AFTER — Rate limited

dotnet run --project tools/Wassal.SessionTests -- session-07-rate-limited

Short alias: session-07. A proof. Happy only when the server shed load and stayed correct and cooperated. PASS when:

• Some 429s (load shed) & some 2xx (not a brick wall)
• Every 429 carried a Retry-After
• DB rows == accepted (rejected did no work)

⚠️ Run the AFTER scenario before adding the limiter and it will (correctly) fail — nothing was shed, so it looks exactly like BEFORE. That's the same rule as every session: match the scenario to the state of the server. Unprotected scenario on the undefended build; rate-limited scenario on the protected build.

---

Task 1 — Flood the Undefended Server (BEFORE) (10 min)

Start the API exactly as it is today — the idempotent server from Lessons 5 & 6, with no rate limiter:

dotnet run --project src/Wassal.Monolith --launch-profile https

Now fire a burst of 30 distinct requests at once and see how many the server accepts:

# From the repo root
dotnet run --project tools/Wassal.SessionTests -- session-07-unprotected-flood

Optional overrides (defaults come from appsettings.json):

dotnet run --project tools/Wassal.SessionTests -- session-07-unprotected-flood `
    --requests 30 `
    --cleanDb true

The console prints a per-request table (with a Retry-After column, all dashes for now) and a summary, then saves a report like Reports/Session-07/Session-07-Unprotected-Flood-RunReport-2026-06-21-143012.txt.

🐛 No backpressure. You'll see 30 accepted, 0 rejected, and 30 orders in the DB. The server shed nothing — it dutifully did the work for every request in the burst. Now imagine the burst is 30,000 from a bot: the server still says "yes" to all of them, until it falls over. Idempotency kept the data correct, but there is no defence against volume. Correct, and on fire.

💡 This scenario passes (exit 0) when the storm gets through — that's the point of a BEFORE diagnostic. "Nothing shed + DB rows == accepted" is the documented undefended state we want to make visible before we fix it.

✅ Done when: the flood shows 0 × 429 and DB rows == accepted == total (the server accepted everything).

---

Task 2 — Add a .NET 8 Token-Bucket Rate Limiter (15 min)

The fix lives entirely on the server, and .NET 8 ships the rate limiter in the framework — no Polly, no NuGet package. We register a single named token bucket and apply it to the orders endpoint.

First, register the limiter in Program.cs (add the usings, then the service):

// src/Wassal.Monolith/Program.cs (additions)
using System.Threading.RateLimiting;
using Microsoft.AspNetCore.RateLimiting;

// ... after builder.Services.AddControllersWithViews(); ...

builder.Services.AddRateLimiter(options =>
{
    // One named token bucket for the orders endpoint.
    options.AddTokenBucketLimiter("orders", o =>
    {
        o.TokenLimit          = 5;                        // bucket holds 5 tokens (max burst)
        o.TokensPerPeriod     = 5;                        // refill 5 tokens...
        o.ReplenishmentPeriod = TimeSpan.FromSeconds(1);  // ...every 1 second
        o.QueueLimit          = 0;                        // don't queue — reject immediately
        o.AutoReplenishment   = true;
    });

    // What the server sends when a request is rejected: 429 + Retry-After.
    options.RejectionStatusCode = StatusCodes.Status429TooManyRequests;
    options.OnRejected = async (context, token) =>
    {
        // The token bucket can estimate when a token will next be free.
        if (context.Lease.TryGetMetadata(MetadataName.RetryAfter, out var retryAfter))
            context.HttpContext.Response.Headers.RetryAfter =
                ((int)retryAfter.TotalSeconds).ToString();

        await context.HttpContext.Response.WriteAsync(
            "Too many requests. Please retry after a moment.", token);
    };
});

Then enable the middleware in the pipeline (after UseRouting, before the endpoints):

// src/Wassal.Monolith/Program.cs (pipeline)
app.UseRouting();

app.UseRateLimiter();   // ← add this line

app.UseAuthorization();

Finally, attach the named limiter to the order-creation action:

// src/Wassal.Monolith/Controllers/OrdersController.cs
using Microsoft.AspNetCore.RateLimiting;   // at the top

// ...

[EnableRateLimiting("orders")]   // ← apply the "orders" bucket to this endpoint
[HttpPost]
public async Task<IActionResult> Create([FromBody] CreateOrderRequest req)
{
    // ... unchanged idempotency logic from Lessons 4 & 5 ...
}

Two things worth noticing:

• QueueLimit = 0 means we reject immediately instead of making callers wait in a queue — that's load shedding, the fail-fast behaviour from Mental Model 1.
• We set RejectionStatusCode to 429 (the default is 503) and add Retry-After in OnRejected — turning a bare rejection into the cooperative contract.

Restart the app so the limiter is live:

dotnet run --project src/Wassal.Monolith --launch-profile https

💡 Why a token bucket and not "X per second" flat? A flat counter either forbids all bursts (bad for normal traffic that clumps) or resets on a hard boundary (letting a double-burst slip through at the edges). The token bucket allows a small, controlled burst (the bucket size) on top of a steady refill rate — the sweet spot for real APIs. We teach this one algorithm only; fixed/sliding windows exist but are out of scope today.

✅ Done when: the app rebuilds with no errors and the orders endpoint is decorated with [EnableRateLimiting("orders")].

---

Task 3 — Prove Load Shedding (AFTER) (10 min)

Run the exact same burst — only the server changed:

# SAME runner, SAME burst engine — different verdict
dotnet run --project tools/Wassal.SessionTests -- session-07-rate-limited

# Short alias:
dotnet run --project tools/Wassal.SessionTests -- session-07

This scenario applies 4 explicit pass/fail checks:

• Load shed (some 429s) — the server rejected the excess
• Some requests still succeeded — it's a limiter, not a brick wall
• Every 429 carried Retry-After — the cooperative contract held
• Rejected did no work (DB rows == accepted) — shed requests created nothing

✅ Sheds load AND stays correct. With a bucket of 5, a burst of 30 yields roughly 5 accepted (200) and ~25 rejected (429), and every one of those 429s carries a Retry-After value in the table. The DB holds exactly as many orders as were accepted — the rejected requests did no work, so there's no half-finished mess to clean up. The server protected itself, served whoever it could, and told the rest precisely when to return.

⚠️ Exact counts will wiggle. Because the bucket refills every second, a slightly slower burst may let a few extra through. The verdict doesn't pin an exact number — it checks the shape: some shed, some served, every 429 cooperative, and the DB matching the accepted count. That's robust across machines.

✅ Done when: all 4 checks PASS — some 429s with Retry-After, some 2xx, and DB rows == accepted.

---

Task 4 — Close the Loop: 429 + Retry-After Meets Backoff (10 min)

Put Lessons 6 and 7 side by side and the whole picture clicks:

SERVER (Lesson 7):  "I'm full. 429. Retry-After: 1."
CLIENT (Lesson 6):  reads Retry-After → waits ~1s → retries
                    with the SAME Idempotency-Key
SERVER:             bucket refilled → 200 OK, replayed=false → 1 order

Three layers, three jobs:
  Idempotency   → the retry is SAFE          (no duplicate order)   [L3–5]
  Backoff+jitter→ the client is GENTLE        (no client-side storm) [L6]
  Rate limiting → the server is PROTECTED     (no server overload)   [L7]

Notice how the pieces depend on each other. Rate limiting causes rejections, which require the client to retry, which is only safe because the server is idempotent, and only gentle because the client backs off — using the Retry-After the server handed it. None of the three lessons stands alone; together they are the real shape of a system that survives a Friday-night surge.

💡 🔜 Continued in Day 7 · Part 2 — The Polite Client Obeys Retry-After. Part 1 (here) taught the server to return 429 + Retry-After. Part 2 teaches the client to read Retry-After, wait at least that long, and retry the same operation with the same Idempotency-Key — turning the 25 rejected requests into eventual successes instead of lost work. Same Day 7, second half.

✅ Done when: you can explain how a single 429 + Retry-After connects all three layers (idempotency, backoff, rate limiting).

---

Task 5 — Capture the Evidence, Tag, Document (5 min)

The BEFORE and AFTER reports under Reports/Session-07/ are your evidence — attach both to the writeup. Then commit the server change + scenarios + session plan and tag the AFTER state:

cd D:\books\distributed-system\Wassal
git add src/Wassal.Monolith/ tools/Wassal.SessionTests/ `
        docs/sessions/Day-07-Rate-Limiting-Load-Shedding.html
git commit -m "feat: Lesson 7 — token-bucket rate limiting + load shedding (429 + Retry-After)"
git tag lesson-07-after

Suggested lesson writeup — docs/lessons/lesson-07-rate-limiting-load-shedding.md:

# Lesson 7 — Rate Limiting & Load Shedding

**Tags:** `lesson-07-before` (unprotected) → `lesson-07-after` (rate limited)
**Builds on:** Lesson 6 (`lesson-06-after`)

## The pain (BEFORE)

With no limiter, a burst is accepted in full — the server has no
backpressure. Idempotency keeps the data correct, but one rude client
can soak all capacity and starve everyone else. Correct, and on fire.

## The fix (AFTER)

A .NET 8 built-in token-bucket limiter (5 tokens, +5/sec, QueueLimit 0)
on the orders endpoint. Excess requests get 429 + Retry-After; the load
is shed fast; accepted requests still each create exactly one order
(rejected requests do no work). Verified by `session-07-rate-limited`.

| Scenario                  | Shed?      | Correctness         | Caller told when? |
|---------------------------|------------|---------------------|-------------------|
| Unprotected flood         | no (0×429) | 1 order/accepted ✅ | no                |
| Rate limited (token bucket)| yes (429) | 1 order/accepted ✅ | Retry-After ✅     |

## The contract

429 + Retry-After is the signal a Lesson-6 client backs off on. Server
sets the pace; client respects it. Idempotency makes the eventual retry
safe.

## What's still deliberately missing (later lessons)

- **The client side of the handshake** — a client that honours Retry-After,
  retries with the same key, and ends with one order. Now covered in
  **Day 7 Part 2 — The Polite Client Obeys Retry-After**.
- **Distributed rate limiting** across replicas (shared counter / Redis) —
  belongs with the scale-out lessons. Today's limiter is per-instance.
- **Per-user / per-API-key limits, quotas, auth** — one global bucket for now.
- **Polly, circuit breaker** — Part 6.
- **Key TTL / expiry** — still parked for the caching lesson.

✅ Done when: both Session-07 reports are saved, the limiter + scenarios + session plan are committed, and lesson-07-after is tagged.

---

Quick Checklist

#	Task	Time	Done When
1	Flood the undefended server → expose no backpressure	10 min	0 × 429, DB rows == accepted == total
2	Add the .NET 8 token-bucket limiter (429 + Retry-After)	15 min	App rebuilds, endpoint has [EnableRateLimiting]
3	Re-run the flood → witness load shedding	10 min	All 4 checks PASS (shed + correct + cooperative)
4	Connect 429 + Retry-After to Lesson 6 backoff	10 min	You can explain all three layers together
5	Save reports, commit, tag lesson-07-after	5 min	Both reports saved + writeup committed

---

🎓 The compounding effect — 7 lessons in. Your order endpoint now survives restarts and body tampering (L4), concurrency (L5), impolite-but-honest retries (L6), and now floods and bots (L7). Idempotency keeps it correct; backoff keeps the client gentle; rate limiting keeps the server standing. That trio — safe retries + client backoff + server backpressure — is exactly how production APIs survive a surge. Same endpoint, seven layers of deliberate, testable defence.

---

➡️ Next — Day 7 · Part 2: The Polite Client Obeys Retry-After

The server now says "not now" with 429 + Retry-After. Part 2 teaches the client to listen: read Retry-After, wait, and retry the same operation with the same Idempotency-Key until it completes.

---

Part of the **Fundamentals of Distributed Systems* series — building Wassal, a distributed food-delivery lab, one concept at a time.*