Day 5 — When Two Requests Race for the Same Key

Lesson 4 made idempotency durable. But what happens when two retries arrive at the same instant, with the same key, and both pass the "is this new?" check before either has written? Today we expose the race, witness it failing, and harden against it.

📘 Where we left off in Lesson 4: You built a durable DB-backed idempotency store with a unique index on Key and a SHA-256 body hash. Same-key retries replay; mismatched bodies get 409. Tagged as lesson-04-after.

At the end we explicitly punted one thing: "two POSTs with the same fresh key, in flight at the exact same moment, can both pass the 'is this key new?' check before either has SaveChanged." Today we tackle that.

The race window in detail: both requests read the IdempotencyKeys table between t₁ and t₂ and find nothing. Whoever calls SaveChanges first wins. The loser either crashes with a DbUpdateException (we currently return 500 — bad UX), or, if there were no unique index, creates a duplicate row (silent corruption). The fix is to catch the loser's exception and replay the winner's response.

🎯 Today's goal: Add a small artificial delay between the "check" and "write" steps of OrdersController to widen the race window, fire 10 simultaneous POSTs with the same idempotency key, witness the race causing 500 errors (Lesson 4's defence-in-depth working at the DB layer but not the API layer), then add a DbUpdateException handler that converts the race-loss into a clean replay — proving idempotency holds under genuine concurrency. Five sequential tasks, ~50 minutes.

---

🧠 Mental Model — How can 10 requests succeed but only 1 order exist?

Same business operation:
Create Order: Koshary + Pizza
Idempotency-Key: ABC-123

10 HTTP requests arrive at nearly the same time.

Important:
10 successful responses does NOT mean 10 orders were created.
It means the API handled all 10 attempts correctly.

This is the most important idea in this lesson. When we say:

10 requests with the same key
10 OK responses
All return the same Order Id
9 have replayed = true
Database row count = 1

We are not saying the system created 10 orders. We are saying the system received 10 attempts for the same logical operation, but it created the order only once. The other attempts received the same result as the first one.

Think of it like this: the first request is the real creator. It creates the order and stores the relationship between the Idempotency-Key and the new Order Id. The other 9 requests are retries of the same intent, so they must not create new orders. They should return the same Order Id.

Request 1:
- Checks key ABC-123
- Key is not found yet
- Creates Order Id = 55
- Stores ABC-123 → Order Id 55
- Returns replayed = false

Request 2:
- Uses the same key ABC-123
- Discovers that another request already created the order
- Reads Order Id = 55
- Returns replayed = true

Request 3:
- Same key ABC-123
- Reads the same winner order
- Returns Order Id = 55
- Returns replayed = true

...

Request 10:
- Same key ABC-123
- Reads the same winner order
- Returns Order Id = 55
- Returns replayed = true

So the final result looks like this:

Request	Creates New Order?	Returned Order Id	Replayed?
1	Yes	55	false
2	No	55	true
3	No	55	true
4	No	55	true
5	No	55	true
6	No	55	true
7	No	55	true
8	No	55	true
9	No	55	true
10	No	55	true

💡 What does replayed=false mean? It means this request created the order for the first time.

💡 What does replayed=true mean? It means this request did not create a new order. It returned the previously created result for the same Idempotency-Key.

⚠️ Why does the database contain only 1 order? Because all 10 requests represent the same business operation. Same key, same body, same intent. Idempotency means we execute that intent once and replay the result for duplicates.

Why is this better than HTTP 500? Before this fix, the database was protected by the unique index, but the API returned errors to the losing requests. After this fix, the losing requests become clean replays. The client gets a successful response instead of a confusing server error.

---

🧪 Verification Discipline — the general rule for every session

Before we touch the tasks, one rule that applies to every lesson in this series, not just Lesson 5. We do not verify our work with throwaway PowerShell snippets that we paste once and forget. PowerShell is fine for a quick poke, but it is not a proof you can trust tomorrow. Every session must have a repeatable .NET scenario runner that lives in the repo under tools/Wassal.SessionTests.

Think of it like this: a PowerShell snippet is a sticky note — it works once, then it is lost. A scenario class is a real witness — anyone can run it, today or in six months, and get the same verdict. One session = one scenario class. Same engine, repeatable, self-documenting.

Each scenario you write must do all of the following:

• Clean the database when needed, so the run starts from a known state (no leftovers from a previous run faking the result).
• Fire the requests in a repeatable way — same count, same key, same body every time, so the run is deterministic.
• Print a readable console table — one row per request plus a summary, so a human can read the outcome at a glance.
• Apply clear pass/fail rules — the scenario itself decides the verdict and sets the exit code. You should not have to eyeball it.
• Save a timestamped report automatically under Reports/Session-XX/ — e.g. Session-05-...-2026-06-11-143012.txt.
• Never overwrite old reports — the timestamp in the filename guarantees every run is kept, so you always have history.
• Make the result easy to attach to the lesson writeup — copy the saved report straight into the docs as evidence.

One engine, many scenarios — different verdicts. The shared mechanics (clean → fire N parallel → count rows → print table → save report) live once in ConcurrentIdempotencyCore. Each scenario reuses that engine but brings its own pass/fail rules. That is the key idea for this lesson: two scenarios can run the exact same 10 parallel POSTs and still expect opposite outcomes, because they are testing two different states of the code.

---

⚖️ Lesson 5 has TWO scenarios with different verdict logic

This is the part beginners trip on, so read it slowly. Both scenarios fire the same 10 parallel POSTs with the same Idempotency-Key. The difference is what counts as success:

① BEFORE — Race exposure

dotnet run --project tools/Wassal.SessionTests -- session-05-race-exposure

Run this before the fix is in. Its job is to prove the bug exists, so it is happy when it sees failures.

Verdict — PASS when:

• At least one request fails / returns HTTP 500
• Database order count stays at 1

Meaning: the DB unique index protected the data, but the API still gave bad UX. The race is real.

② AFTER — Concurrency-safe replay

dotnet run --project tools/Wassal.SessionTests -- session-05-concurrency-safe-replay
# Short alias:
dotnet run --project tools/Wassal.SessionTests -- session-05

Run this after the fix is in. Its job is to prove the handler works, so it is happy only when every request is clean.

Verdict — PASS when:

• 10 OK responses, 0 failures
• 1 unique Order Id
• DB row count = 1
• replayed = true for the race losers

⚠️ Same input, opposite verdicts — that is intentional. If you ran the BEFORE scenario after the fix, it would "fail" (no 500s to find) — and that failure is actually good news, it means the bug is gone. Always match the scenario to the state of the code: race-exposure on the broken build, concurrency-safe-replay on the fixed build.

---

Task 1 — Expose the Race Condition (15 min)

On a fast local machine the race window is usually too narrow to hit reliably. We're going to deliberately widen it by inserting a 300 ms delay right between the "check" and the "write" — like slow-motion footage of the bug.

Open OrdersController.cs and add ONE line inside the Create method, right after the existing-key check (the if (seen is not null) block) and BEFORE creating the new Order:

src/Wassal.Monolith/Controllers/OrdersController.cs (temporary diagnostic):

// ── 1. If we've seen this key, check & replay
if (key is not null)
{
    var seen = await _db.IdempotencyKeys
        .FirstOrDefaultAsync(k => k.Key == key);

    if (seen is not null) { /* ... existing replay/conflict logic ... */ }
}

// 🐛 RACE-WINDOW WIDENER — REMOVE in Task 5 after the fix is in.
// Simulates a slow DB or network — gives concurrent requests time to
// all pass the check above before any of them writes below.
await Task.Delay(300);

// ── 2. Brand-new intent → create order + record the key (atomically)
var order = new Order { /* ... existing code ... */ };
// ...

Restart the app:

dotnet run --project src/Wassal.Monolith --launch-profile https

Now fire 10 simultaneous POSTs with the SAME idempotency key.

🛠️ One unified test runner — no PowerShell needed. The repo includes a .NET 8 console app at tools/Wassal.SessionTests/ that replaces every PowerShell verification script. Each session = one C# scenario class. Mechanics live in ConcurrentIdempotencyCore (clean DB → fire N parallel via Task.WhenAll → count rows → print table + summary). Configurable via appsettings.json or CLI flags (--baseUrl, --parallel, --key, --cleanDb, --connectionString). Reports are auto-saved with timestamps to Reports/Session-XX/ so previous runs are never overwritten.

Run the race-exposure scenario:

# From the repo root
dotnet run --project tools/Wassal.SessionTests -- session-05-race-exposure

Optional overrides (defaults come from appsettings.json):

dotnet run --project tools/Wassal.SessionTests -- session-05-race-exposure `
    --baseUrl https://localhost:7126 `
    --parallel 10 `
    --cleanDb true

The console prints a per-request table + summary, then writes a timestamped report like Reports/Session-05/Session-05-Race-Exposure-RunReport-2026-06-11-143012.txt. The scenario exits 0 when the documented BEFORE state is observed (≥1 failure + DB count = 1).

🐛 The race in action. You'll see something like 1 OK + 9 FAIL (Status 500), OR sometimes 2-3 OKs and several FAILs. The first request to call SaveChanges wins; the rest hit the unique-index on IdempotencyKeys.Key and the unhandled DbUpdateException bubbles up as a 500. The dedup guarantee technically holds (DB count = 1) but the client experience is terrible — most retries see a server error instead of a clean replay.

⚠️ Two race outcomes are possible depending on timing: (a) only the winner saw "key new", everyone else's check returned the winner's record → all 10 OK + 9 replayed; or (b) several requests passed the check before the winner committed → 1 OK + 9 FAILs as above. On a fast machine with our 300 ms delay you should see (b) dominate.

Tag this broken state:

cd D:\books\distributed-system\Wassal
git add src/Wassal.Monolith/
git commit -m "diagnostic: Lesson 5 — widen race window with Task.Delay to expose the bug"
git tag lesson-05-before

✅ Done when: you witnessed FAIL responses or duplicate behaviour, and tag lesson-05-before exists.

---

Task 2 — Understand What Just Happened (5 min)

Read this once. It's the entire mental model in 4 bullets:

1. The check & the write are two separate operations. Between them, anything can happen. That gap is the race window.
2. Concurrency multiplies that gap. 10 requests can all be inside the same gap at once, all having read "key doesn't exist yet."
3. The DB unique index saves the data. Even if 10 requests try to insert the same key, the index lets only one succeed. The other 9 throw DbUpdateException. That's the defence we already built in Lesson 4.
4. But we're not handling the exception. ASP.NET Core's default is to return 500. Our clients see errors. The data is safe but the UX is broken. That's what we fix next.

💡 The general pattern. "Check-then-act" is racy whenever check and act are on shared state and not atomic. The cure is either: (a) make check+act atomic (a single DB query with INSERT ... ON CONFLICT, or a row-level lock), or (b) let the act fail and recover from the failure (catch the unique violation, look up the winner's row, return its result). Today we're doing option (b) because it composes well with EF Core.

✅ Done when: you can explain in one sentence why the race happens and why the unique index alone isn't enough.

---

Task 3 — Catch DbUpdateException → Replay the Winner (15 min)

Replace the order-creation block (everything from // ── 2. Brand-new intent through the final return Ok(...)) with the version below. We wrap SaveChangesAsync in try/catch and, on a unique-key violation, re-query for the winner's record and replay its order.

src/Wassal.Monolith/Controllers/OrdersController.cs (replace the lower half of Create):

// ── 2. Brand-new intent → create order + record the key (atomically)
var order = new Order
{
    RestaurantId = req.RestaurantId,
    Items        = req.Items,
    TotalAmount  = req.TotalAmount,
    CreatedAt    = DateTime.UtcNow,
};
_db.Orders.Add(order);

if (key is not null)
{
    _db.IdempotencyKeys.Add(new IdempotencyKey
    {
        Key             = key,
        RequestBodyHash = bodyHash,
        Order           = order,
    });
}

try
{
    await _db.SaveChangesAsync();
    return Ok(new OrderResponse(order.Id, order.RestaurantId, order.TotalAmount,
                                Replayed: false));
}
catch (DbUpdateException ex) when (IsUniqueKeyViolation(ex) && key is not null)
{
    // Another request committed the same key first. Discard our work-in-progress
    // entities, fetch the winner's record, and replay it. This is the race-loser's
    // happy path.
    _db.ChangeTracker.Clear();

    var winner = await _db.IdempotencyKeys
        .AsNoTracking()
        .FirstOrDefaultAsync(k => k.Key == key);

    if (winner is null)
        throw; // shouldn't happen, but if it does we surface it loudly

    // Body-hash safety check — same as the cold-path
    if (winner.RequestBodyHash != bodyHash)
        return Conflict(new { error = "Idempotency-Key was used with a different body." });

    var winnerOrder = await _db.Orders.FindAsync(winner.OrderId);
    if (winnerOrder is null)
        throw; // referential integrity is broken — surface it

    Response.Headers["Idempotency-Replayed"] = "true";
    return Ok(new OrderResponse(winnerOrder.Id, winnerOrder.RestaurantId,
                                winnerOrder.TotalAmount, Replayed: true));
}

// Helper: detects SQL Server's unique-index violation (error numbers 2601, 2627)
static bool IsUniqueKeyViolation(DbUpdateException ex) =>
    ex.InnerException is Microsoft.Data.SqlClient.SqlException sql
    && (sql.Number == 2601 || sql.Number == 2627);

Two important details:

• _db.ChangeTracker.Clear() wipes the in-flight (uncommitted) Order and IdempotencyKey from EF's tracker, so the next query reads cleanly from the DB.
• The when clause on the catch filters narrowly — we only intercept the specific exception type we know how to handle. Anything else (connection errors, schema mismatches, real bugs) still propagates as 500. Defence-in-depth, not blanket suppression.

Restart the app:

dotnet run --project src/Wassal.Monolith --launch-profile https

💡 SQL Server unique-violation error numbers. 2601 = "duplicate key with a unique index", 2627 = "violation of unique constraint." Both mean the same thing in practice — we check for either. For PostgreSQL the equivalent is SQLSTATE 23505 on PostgresException. The pattern is identical, only the error code changes.

✅ Done when: the app rebuilds with no errors and listens on https://localhost:7126.

---

Task 4 — Demonstrate the Fix Under Real Concurrency (10 min)

Re-run the exact same scenario from Task 1 — the 300 ms delay is still in the controller, so the race window is still wide open:

# SAME runner, SAME execution engine — different scenario class, different verdict logic
dotnet run --project tools/Wassal.SessionTests -- session-05-concurrency-safe-replay

# Short alias:
dotnet run --project tools/Wassal.SessionTests -- session-05

This scenario uses the same ConcurrentIdempotencyCore as Task 1 — same clean, same 10 parallel POSTs, same key. The only difference is that this scenario class applies 5 explicit pass/fail checks on the result:

• OK == total — every request returned 2xx
• FAIL == 0 — no HTTP errors
• Unique ids == 1 — all responses point at one order
• DB row count == 1 — only one order actually written
• Replayed == total - 1 — the winner created, all losers replayed

Exit code 0 means all five passed → idempotency holds under concurrency. Non-zero means the DbUpdateException fix isn't in place yet (run the race-exposure scenario to confirm).

✅ Idempotency holds under concurrency. Read this result the way the Mental Model section above explains it: All 10 OK means every HTTP response succeeded — not that 10 orders were created. 1 unique id means all 10 responses refer to the same single order. DB row count = 1 means only one order was actually inserted. Behind that: 9 responses carry replayed=true. The first race-winner created the order; the other 9 lost the race, caught the unique-violation, looked up the winner's record, and replayed it. The client sees a uniform success.

✅ Done when: all 10 jobs returned OK with the same id, ~9 have replayed=true, and DB has exactly 1 row.

---

Task 5 — Remove the Diagnostic Delay, Tag, Document, Push (5 min)

Open OrdersController.cs and delete the await Task.Delay(300); line you added in Task 1. The race window narrows back to its natural ~microsecond size — but the fix still works. Re-run the Task 4 scenario one more time to confirm.

Commit, tag, and create the writeup:

cd D:\books\distributed-system\Wassal
git add src/Wassal.Monolith/
git commit -m "feat: Lesson 5 — handle DbUpdateException to make idempotency concurrency-safe"
git tag lesson-05-after

docs/lessons/lesson-05-concurrent-idempotency.md:

# Lesson 5 — Concurrent Idempotency

**Tags:** `lesson-05-before` → `lesson-05-after`
**Builds on:** Lesson 4 (`lesson-04-after`)

## The pain (BEFORE)

With a 300 ms artificial delay between the check and the write, firing
10 parallel POSTs with the same key gave us 1 OK + 9 HTTP 500 errors.
The data was protected by the unique index on `IdempotencyKeys.Key`,
but the API layer didn't recognise the loser's exception and returned
500. Clients saw failures even though the operation had succeeded.

## The fix (AFTER)

Wrap `SaveChangesAsync` in `try/catch`, narrow the `catch` to
`DbUpdateException` whose inner SQL exception has code 2601 or 2627
(unique-violation). On match: clear the change tracker, re-fetch the
winner's record, and return it with `replayed: true`. The race-loser
now has a happy path — same shape as a normal retry.

| Scenario                                       | Before          | After                          |
|------------------------------------------------|-----------------|---------------------------------|
| 10 concurrent same-key POSTs                   | 1 OK + 9 × 500  | 10 OK, 1 unique id, 9 replayed |
| DB rows created                                | 1               | 1                              |
| Client error rate                              | 90%             | 0%                             |

## What's still deliberately missing

- **Optimistic vs pessimistic.** We chose optimistic (let it fail, recover).
  Pessimistic alternative: row-level lock or `INSERT ... ON CONFLICT`.
  When we move to PostgreSQL or introduce Redis-backed dedup we'll
  revisit.
- **Retries beyond N.** A pathological client sending 10K parallel
  requests with the same key will hammer SQL Server. Rate limiting at
  the gateway layer is the right defence — a later lesson.
- **Observability.** We can't see how often the race fires today.
  Logging counters live in Lesson 24 (Observability).

## Skills reinforced

- Idempotency-Key + DB-backed dedup (Lessons 3, 4)
- EF Core SaveChangesAsync atomic writes (Lesson 4)
- Before/After tagging + writeup (Lessons 3, 4)
- PowerShell HTTPS verification

## Skills added

- Recognising "check-then-act" race conditions
- Catching `DbUpdateException` for unique violations
- Defence-in-depth (app check + DB constraint + exception handler)
- Parallel HTTP from PowerShell with `Start-Job`
- Deliberately inserting latency to expose timing bugs

git add docs/lessons/lesson-05-concurrent-idempotency.md `
        docs/sessions/Day-05-Concurrent-Idempotency.html
git commit -m "docs: Lesson 5 writeup + Day 5 session plan"
git push origin master --tags

git tag
# expected:
#   lesson-01-before
#   lesson-01-pain-documented
#   lesson-02-after
#   lesson-04-before
#   lesson-04-after
#   lesson-05-before          ← new
#   lesson-05-after           ← new
#   v0-monolith-baseline

✅ Done when: the delay is removed, 8 tags exist on GitLab, and the writeup is committed.

---

Quick Checklist

#	Task	Time	Done When
1	Add 300 ms delay + parallel POSTs → expose race	15 min	FAIL responses observed + tag lesson-05-before
2	Mental model of check-then-act races	5 min	You can explain the race in one sentence
3	Catch DbUpdateException + replay winner	15 min	App rebuilds with no errors
4	Re-run parallel scenario → witness the fix	10 min	10 OK + 1 unique id + 9 replayed + 1 DB row
5	Remove delay, tag lesson-05-after, push, document	5 min	8 tags on GitLab + writeup md

---

🎓 The compounding effect — 5 lessons in. You now have an idempotent POST endpoint that survives app restarts (Lesson 4), defends against body tampering (Lesson 4), and stays correct under genuine concurrency (Lesson 5). The same controller method handles all three. Every line of defence layers on the previous one. That's how production-grade systems are actually written — never in one shot, always in layered, deliberate, testable iterations.

---

Part of the **Fundamentals of Distributed Systems* series — building Wassal, a distributed food-delivery lab, one concept at a time.*