Day 4 — Idempotency, but Production-Grade: Surviving the Restart

Lesson 3's ConcurrentDictionary worked beautifully — until you restarted the app. Today we replace it with a durable DB-backed store and add body-hash validation. Same pattern, real-world strength.

📘 Where we left off in Lesson 3: You built a POST /api/orders endpoint that used an Idempotency-Key header to dedupe retries via an in-memory ConcurrentDictionary<string, int>. Same key sent twice → one row. Tagged as lesson-02-after.

We listed "what's deliberately missing" at the end:

• Storage is in-memory — lost on app restart, not shared across replicas
• No body validation — same key with a different body silently returns the first answer
• No TTL — keys live forever in memory

Today we tackle the first two. (TTL we'll handle in a later lesson together with cache eviction.)

🎯 Today's goal: Replace the in-memory idempotency map with a durable IdempotencyKeys table, add request-body hashing so the server can detect "same key, different body" abuse, and prove both behaviours with end-to-end PowerShell scenarios — culminating in tags lesson-04-before (restart-wipe pain) and lesson-04-after (persistent + body-validated). Six sequential tasks, ~55 minutes.

---

🧠 Mental Model — What is the Idempotency-Key unique for?

A common misunderstanding is to think that the Idempotency-Key is unique per user session. That is not accurate.

The better mental model is: one Idempotency-Key per business operation / request intent.

For example, if the user is creating one order, the client generates one key for that specific create-order intent:

Create Order: Koshary + Pizza  →  Idempotency-Key = ABC-123

If the same request is retried because of a timeout, double click, network issue, or client retry logic, the client must reuse the same key:

First attempt        →  Key = ABC-123
Retry for same order →  Key = ABC-123
Retry again          →  Key = ABC-123

This tells the server: "These are not different orders. They are the same operation being repeated."

But if the user wants to create another order, the client must generate a new key:

Create Order: Koshary + Pizza  →  Key = ABC-123
Create Order: Burger           →  Key = XYZ-999

💡 Important: It is not one key per session, and it is not one new key per HTTP retry. It is one key per logical operation. Retries of the same operation reuse the same key. New operations get new keys.

So when we say:

10 requests with the same key
10 successful responses
All return the same Order Id
9 of them have replayed = true
Database row count = 1

This does not mean the system created 10 orders. It means the API returned successful responses for all 10 attempts, but only the first request created the order. The other 9 requests simply replayed the result of the first one.

That is the real goal of idempotency: same operation, same key, same result — without creating duplicates.

Note: what happens when several of these requests arrive at the same instant (true concurrency) is a separate concern — we handle that in Lesson 5.

---

Task 1 — Demonstrate the In-Memory Pain (5 min)

Before we change anything, let's feel the limitation of the in-memory store. Start with a clean Orders table and the app running on https://localhost:7126:

# Wipe orders to start clean
sqlcmd -S "(local)" -E -d WassalDb -Q "DELETE FROM Orders;"

# Send a POST with a fresh idempotency key
$body = '{"restaurantId":1,"items":"Koshary,Pizza","totalAmount":95.50}'
$key  = [guid]::NewGuid().ToString()
$headers = @{ "Content-Type" = "application/json"; "Idempotency-Key" = $key }

$r1 = Invoke-RestMethod -Uri "https://localhost:7126/api/orders" `
        -Method POST -Body $body -Headers $headers
"First call:  id=$($r1.id)  replayed=$($r1.replayed)"

Now stop and restart the app (Ctrl+C in the dotnet window, then dotnet run --project src/Wassal.Monolith --launch-profile https). Wait for it to come back up, then in PowerShell re-use the same $key:

# Reuse the SAME key. We expect the server to remember and replay #1.
$r2 = Invoke-RestMethod -Uri "https://localhost:7126/api/orders" `
        -Method POST -Body $body -Headers $headers
"Second call (after restart):  id=$($r2.id)  replayed=$($r2.replayed)"

# How many rows in DB?
(Invoke-RestMethod -Uri "https://localhost:7126/api/orders").Count

🐛 The pain: $r2.id is different from $r1.id, replayed=false, and the database has 2 rows. The restart wiped the ConcurrentDictionary, so the second call looked like a brand-new intent. In production this happens every deploy, every crash, every auto-scale event. Unacceptable.

Capture this state as our starting point for today:

cd D:\books\distributed-system\Wassal
git tag lesson-04-before

✅ Done when: you've witnessed the double-row after restart and tag lesson-04-before exists.

---

Task 2 — Add the IdempotencyKey Entity + Migration (10 min)

We need a place to durably remember key → orderId. A new table is the simplest fit.

src/Wassal.Monolith/Models/IdempotencyKey.cs:

namespace Wassal.Monolith.Models;

public class IdempotencyKey
{
    public int Id { get; set; }
    public string Key { get; set; } = string.Empty;             // the client-provided GUID
    public string RequestBodyHash { get; set; } = string.Empty; // SHA-256 hex of the request body
    public int OrderId { get; set; }                            // FK to Order
    public Order? Order { get; set; }                           // nav property (lets EF set OrderId in one SaveChanges)
    public DateTime CreatedAt { get; set; } = DateTime.UtcNow;
}

Register it in WassalDbContext with a unique index on Key (so the DB itself enforces "one record per key"):

src/Wassal.Monolith/Data/WassalDbContext.cs (additions):

// Add next to the existing DbSet declarations:
public DbSet<IdempotencyKey> IdempotencyKeys => Set<IdempotencyKey>();

// Inside OnModelCreating, AFTER the existing Restaurant seed, add:
b.Entity<IdempotencyKey>()
    .HasIndex(k => k.Key)
    .IsUnique();

Create and apply the migration:

cd D:\books\distributed-system\Wassal
dotnet ef migrations add AddIdempotencyKeysTable --project src/Wassal.Monolith
dotnet ef database update --project src/Wassal.Monolith

💡 Why a unique index? Two requests with the same key arriving at the same instant could both pass the "is this key new?" check before either has written. The DB-level unique index turns that race into a guaranteed DbUpdateException we can catch and treat as "key already exists" — defense in depth beyond the application-level check.

✅ Done when: an empty IdempotencyKeys table exists in WassalDb with a unique index on Key.

---

Task 3 — Replace ConcurrentDictionary with the DB Store (15 min)

Replace the entire contents of OrdersController.cs with this DB-backed version. The shape is the same as Lesson 3 — only the storage changed:

src/Wassal.Monolith/Controllers/OrdersController.cs (replace):

using System.Security.Cryptography;
using System.Text;
using System.Text.Json;
using Microsoft.AspNetCore.Mvc;
using Microsoft.EntityFrameworkCore;
using Wassal.Monolith.Data;
using Wassal.Monolith.Models;

namespace Wassal.Monolith.Controllers;

[ApiController]
[Route("api/[controller]")]
public class OrdersController : ControllerBase
{
    private readonly WassalDbContext _db;
    public OrdersController(WassalDbContext db) => _db = db;

    public record CreateOrderRequest(int RestaurantId, string Items, decimal TotalAmount);
    public record OrderResponse(int Id, int RestaurantId, decimal TotalAmount, bool Replayed);

    [HttpPost]
    public async Task<IActionResult> Create([FromBody] CreateOrderRequest req)
    {
        var key      = Request.Headers["Idempotency-Key"].FirstOrDefault();
        var bodyHash = ComputeSha256(JsonSerializer.Serialize(req));

        // ── 1. If we've seen this key, check & replay
        if (key is not null)
        {
            var seen = await _db.IdempotencyKeys
                .FirstOrDefaultAsync(k => k.Key == key);

            if (seen is not null)
            {
                // Same key with a DIFFERENT body → reject (will fully explore in Task 5)
                if (seen.RequestBodyHash != bodyHash)
                    return Conflict(new { error = "Idempotency-Key was used with a different body." });

                // Same key + same body → return cached
                var cachedOrder = await _db.Orders.FindAsync(seen.OrderId);
                if (cachedOrder is not null)
                {
                    Response.Headers["Idempotency-Replayed"] = "true";
                    return Ok(new OrderResponse(
                        cachedOrder.Id, cachedOrder.RestaurantId, cachedOrder.TotalAmount,
                        Replayed: true));
                }
            }
        }

        // ── 2. Brand-new intent → create order + record the key (atomically)
        var order = new Order
        {
            RestaurantId = req.RestaurantId,
            Items        = req.Items,
            TotalAmount  = req.TotalAmount,
            CreatedAt    = DateTime.UtcNow,
        };
        _db.Orders.Add(order);

        if (key is not null)
        {
            _db.IdempotencyKeys.Add(new IdempotencyKey
            {
                Key             = key,
                RequestBodyHash = bodyHash,
                Order           = order,  // EF resolves OrderId after Order insert
            });
        }

        await _db.SaveChangesAsync(); // one transaction → both rows or neither

        return Ok(new OrderResponse(order.Id, order.RestaurantId, order.TotalAmount,
                                    Replayed: false));
    }

    [HttpGet]
    public async Task<IActionResult> List() =>
        Ok(await _db.Orders.OrderByDescending(o => o.CreatedAt).ToListAsync());

    private static string ComputeSha256(string input)
    {
        var bytes = SHA256.HashData(Encoding.UTF8.GetBytes(input));
        return Convert.ToHexString(bytes);
    }
}

Notice three differences from Lesson 3's controller:

• The static ConcurrentDictionary is gone. State now lives in the database.
• The constructor injects WassalDbContext exactly as before, but we use it for both orders and idempotency keys.
• Both inserts happen in a single SaveChangesAsync — EF wraps that in a transaction, so it's all-or-nothing.

Stop and restart the app:

dotnet run --project src/Wassal.Monolith --launch-profile https

✅ Done when: the app rebuilds with no errors and listens on https://localhost:7126.

---

Task 4 — Demonstrate Persistence Across Restarts (5 min)

Repeat the exact scenario from Task 1 — but expect a different outcome this time.

# Clean slate again
sqlcmd -S "(local)" -E -d WassalDb -Q "DELETE FROM Orders; DELETE FROM IdempotencyKeys;"

# Send first POST with a fresh key
$body = '{"restaurantId":1,"items":"Koshary,Pizza","totalAmount":95.50}'
$key  = [guid]::NewGuid().ToString()
$headers = @{ "Content-Type" = "application/json"; "Idempotency-Key" = $key }

$r1 = Invoke-RestMethod -Uri "https://localhost:7126/api/orders" `
        -Method POST -Body $body -Headers $headers
"First call:  id=$($r1.id)  replayed=$($r1.replayed)"

Now restart the app again (Ctrl+C, then re-run), and re-send with the SAME key:

$r2 = Invoke-RestMethod -Uri "https://localhost:7126/api/orders" `
        -Method POST -Body $body -Headers $headers
"Second call (after restart):  id=$($r2.id)  replayed=$($r2.replayed)"

(Invoke-RestMethod -Uri "https://localhost:7126/api/orders").Count

✅ The fix: $r2.id equals $r1.id, replayed=true, and only ONE row exists in Orders. The DB-backed store survived the restart. Production-grade.

✅ Done when: after-restart retry returns the same id with replayed=true and DB has 1 order row.

---

Task 5 — Verify the Body-Hash Defence (409 Conflict on mismatch) (10 min)

The controller already returns 409 Conflict when the same key is used with a different body — let's prove it.

# Reuse the same key from Task 4, but send a DIFFERENT body
$differentBody = '{"restaurantId":2,"items":"Burger","totalAmount":50.00}'

try {
    Invoke-RestMethod -Uri "https://localhost:7126/api/orders" `
        -Method POST -Body $differentBody -Headers $headers
} catch {
    "Status: $($_.Exception.Response.StatusCode.value__)"
    $reader = New-Object System.IO.StreamReader($_.Exception.Response.GetResponseStream())
    "Body:   $($reader.ReadToEnd())"
}

✅ Expected output: Status 409 and a JSON body like {"error":"Idempotency-Key was used with a different body."}. The server refused to silently overwrite the first intent. The DB still has exactly 1 order row.

This catches a real-world bug: a flaky client that mutates the body between retries (different shipping address, different total) but accidentally re-uses the key. Better to fail loud than silently lose data.

💡 Why a hash, not the raw body? Bodies can be huge (think file uploads, large order payloads). Storing a 32-byte SHA-256 hash is constant cost and still gives strong "same body or not" detection. Collisions for two genuinely-different bodies are astronomically unlikely.

✅ Done when: the mismatched-body request returns 409 and the DB still has only 1 order row.

---

Task 6 — Tag, Document, Push (10 min)

Commit the new code, tag the AFTER state, and push to GitLab:

cd D:\books\distributed-system\Wassal
git add src/Wassal.Monolith/
git status
git commit -m "feat: Lesson 4 — DB-backed idempotency store + body-hash validation"
git tag lesson-04-after

Create a short writeup capturing the before/after evidence:

docs/lessons/lesson-04-persistent-idempotency.md:

# Lesson 4 — Persistent Idempotency + Body Validation

**Tags:** `lesson-04-before` → `lesson-04-after`
**Date:** 2026-05-20
**Builds on:** Lesson 3 (`lesson-02-after`)

## The pain (BEFORE)

In-memory `ConcurrentDictionary<string, int>` survives requests but NOT
restarts. Every deploy, crash, or auto-scale event silently breaks the
dedup guarantee — clients that retry across a restart get double-charged.

| Scenario                          | Result                       |
|-----------------------------------|------------------------------|
| Same key, no restart              | dedup works ✅                |
| Same key, AFTER app restart       | creates new order ❌          |

## The fix (AFTER)

Move the key→orderId map into the `IdempotencyKeys` table (unique index
on `Key`). Add a SHA-256 hash of the request body to detect "same key,
different body" abuse. Both writes happen in one `SaveChangesAsync` so
the order and the idempotency record are atomic.

| Scenario                          | Result                                |
|-----------------------------------|---------------------------------------|
| Same key + same body              | dedup, `replayed=true` ✅              |
| Same key + same body, post-restart| dedup, `replayed=true` ✅              |
| Same key + DIFFERENT body         | `409 Conflict` ✅                      |
| Different key                     | new order, `replayed=false` ✅         |

## What's still deliberately missing (next lessons)

- **TTL on keys.** Today they live forever. Real systems expire records
  after 24h to 7d. Will be combined with cache eviction in a later lesson.
- **Shared across instances.** Two app replicas share the SAME DB, so
  this design already works for horizontal scale. Bonus skill unlocked
  for free.
- **Race condition under concurrent identical requests.** Two POSTs with
  the same fresh key, in flight at the exact same moment, can both pass
  the "is this key new?" check before either has SaveChanged. The unique
  index turns the second insert into a `DbUpdateException` we should
  catch and treat as "key already exists, replay." We'll do that
  explicitly when we get to concurrency-control lessons.

## Skills reinforced from Lesson 3

- POST endpoint design with `[ApiController]` + `ControllerBase`
- `Idempotency-Key` header pattern
- Before → After tagging discipline
- PowerShell HTTPS verification scenarios

## Skills added today

- EF Core entity + unique index migration
- SHA-256 body hashing
- Returning HTTP `409 Conflict` for idempotency abuse
- Atomic multi-entity writes via a single `SaveChangesAsync`

Commit the writeup and push everything to GitLab:

git add docs/lessons/lesson-04-persistent-idempotency.md `
        docs/sessions/Day-04-Persistent-Idempotency.html
git commit -m "docs: Lesson 4 writeup + Day 4 session plan"
git push origin master --tags

Verify the full tag history:

git tag
# expected:
#   lesson-01-before
#   lesson-01-pain-documented
#   lesson-02-before
#   lesson-02-after
#   lesson-04-before          ← new
#   lesson-04-after           ← new
#   v0-monolith-baseline

✅ Done when: 7 tags on local + GitLab, and lesson-04-persistent-idempotency.md captures the before/after evidence.

---

Quick Checklist

#	Task	Time	Done When
1	Demo in-memory pain (restart wipes dedup)	5 min	2 rows after restart + tag lesson-04-before
2	Add IdempotencyKey entity + migration	10 min	Empty IdempotencyKeys table with unique index
3	Replace ConcurrentDictionary with DB store	15 min	App rebuilds with no errors
4	Demo persistence across restart	5 min	Same id, replayed=true, 1 row after restart
5	Verify body-hash defence (409 on mismatch)	10 min	Different body → HTTP 409, still 1 row
6	Document, commit, push, tag lesson-04-after	10 min	7 tags on GitLab + writeup md

---

🎓 The compounding effect: Look at what you can now do that you couldn't 3 days ago: write an idempotent POST endpoint, back it with a durable store, defend against client misuse with body hashing, and tag every step so you can git diff a year from now and still understand the journey. Each lesson keeps the previous lesson's skills active — that's what consolidation looks like in practice.

---

Part of the **Fundamentals of Distributed Systems* series — building Wassal, a distributed food-delivery lab, one concept at a time.*