Day 3 — REST & the Double-Charge Bug: Making POST Idempotent

Every developer has accidentally double-clicked. Every distributed system retries on its own. Both create duplicates — unless your POST endpoint is idempotent. Today we feel the bug, then fix it.

By the end of this session you'll have a naive POST /api/orders that double-charges on retry (tagged lesson-02-before), then an idempotent version that dedupes retries by key (tagged lesson-02-after) — with real database rows proving the difference.

📚 What is idempotency? (60-second read)

Idempotent = doing the operation 100 times has the same effect as doing it once. The math definition is dry (a function where f(f(x)) = f(x)) but the everyday version is much simpler.

🛗 Elevator button. Press it once → elevator is called. Press it 47 more times → still one elevator coming. Same result either way. That's idempotency.

Some HTTP methods are idempotent by nature. POST is the troublemaker:

Method	Idempotent?	Why
GET /orders/1	✅	Just reads. Calling 100× still returns the same order.
PUT /orders/1 {status:"paid"}	✅	Sets to a specific value. 100 sets = 1 set.
DELETE /orders/1	✅	Already gone is still gone.
POST /orders {pizza}	❌	Creates a new row every time it's called.

And here's the catch: networks lose responses constantly. The client doesn't know whether the server got the POST. So it retries. So the server creates again. So the customer is charged twice.

🐛 Without idempotency — the bug. The client sends POST /orders { pizza, $95.50 }. The server creates Order #1 and charges the customer — but the response is lost on the wire (timeout). The client, seeing no reply, retries the same body with no key. The server has no way to tell this is a retry, so it creates Order #2 and charges again. The customer is charged TWICE for ONE intent.

✅ With an Idempotency-Key — the fix. The client sends the same POST but adds Idempotency-Key: ABC-123. The server checks its ledger: key not seen → it creates Order #1 and records { ABC-123 → #1 }. The response is lost again, the client retries with the same key, and this time the server finds the key already in its ledger — so it replays Order #1 with { id: 1, replayed: true } and does no new work. The customer is charged ONCE — the retry is now safe.

The pattern in one line: the client picks a unique ID per intent, the server remembers what it did for that ID, and any retry with the same ID gets the cached answer instead of triggering new work.

That's the whole idea. The rest of today is just wiring it up to Wassal and proving the difference with real database rows.

💡 BEFORE (tag: lesson-02-before) — Naive POST endpoint: the same request body sent twice → two rows in the Orders table. The server has no way to tell intent #2 from intent #1.

AFTER (tag: lesson-02-after) — Idempotency-Key check: the same key sent twice → one row in Orders. The second response is the cached result of the first, with header Idempotency-Replayed: true.

---

Task 1 — Add the Order Model + Migration (10 min)

Wassal v0 only knows about Restaurants so far. To demonstrate the double-charge bug we need an Order entity and a DB table to count rows in.

Create the model:

// src/Wassal.Monolith/Models/Order.cs
namespace Wassal.Monolith.Models;

public class Order
{
    public int Id { get; set; }
    public int RestaurantId { get; set; }
    public string Items { get; set; } = string.Empty; // simplified: comma-separated names
    public decimal TotalAmount { get; set; }
    public DateTime CreatedAt { get; set; } = DateTime.UtcNow;
}

Register it in the DbContext. Open WassalDbContext.cs and add this property next to the existing Restaurants:

// src/Wassal.Monolith/Data/WassalDbContext.cs (additions)
public DbSet<Order> Orders => Set<Order>();

Create the migration and apply it:

cd D:\books\distributed-system\Wassal
dotnet ef migrations add AddOrdersTable --project src/Wassal.Monolith
dotnet ef database update --project src/Wassal.Monolith

✅ Done when: a new Orders table exists in WassalDb (verify via SSMS — should be empty).

---

Task 2 — Add the Naive OrdersController (no idempotency yet) (10 min)

This is the version with the bug — every POST creates a new Order, no matter what. This is the BEFORE state.

// src/Wassal.Monolith/Controllers/OrdersController.cs
using Microsoft.AspNetCore.Mvc;
using Microsoft.EntityFrameworkCore;
using Wassal.Monolith.Data;
using Wassal.Monolith.Models;

namespace Wassal.Monolith.Controllers;

[ApiController]
[Route("api/[controller]")]
public class OrdersController : ControllerBase
{
    private readonly WassalDbContext _db;
    public OrdersController(WassalDbContext db) => _db = db;

    public record CreateOrderRequest(int RestaurantId, string Items, decimal TotalAmount);
    public record OrderResponse(int Id, int RestaurantId, decimal TotalAmount);

    [HttpPost]
    public async Task<IActionResult> Create([FromBody] CreateOrderRequest req)
    {
        var order = new Order
        {
            RestaurantId  = req.RestaurantId,
            Items         = req.Items,
            TotalAmount   = req.TotalAmount,
            CreatedAt     = DateTime.UtcNow,
        };
        _db.Orders.Add(order);
        await _db.SaveChangesAsync();

        return Ok(new OrderResponse(order.Id, order.RestaurantId, order.TotalAmount));
    }

    [HttpGet]
    public async Task<IActionResult> List() =>
        Ok(await _db.Orders.OrderByDescending(o => o.CreatedAt).ToListAsync());
}

💡 Why ControllerBase instead of Controller? ASP.NET Core gives you two base classes for controllers:

• Controller — for traditional MVC pages that return HTML views. Carries View(), PartialView(), ViewBag, ViewData, TempData, and other view-related machinery.
• ControllerBase — for APIs that return JSON or status codes. Lighter — no View baggage, just Ok(), BadRequest(), NotFound(), Created(), etc.

OrdersController is a pure JSON API — every action returns Ok(...) with serialized data, never an HTML view. ControllerBase matches that intent in one keyword: it tells anyone reading the class "this is an API endpoint, not a page." Notice that HomeController from Day 1 still inherits from Controller — because it actually returns View(restaurants) to render Views/Home/Index.cshtml. Pick the base class that matches what the class does: page = Controller, API = ControllerBase. The [ApiController] attribute on top reinforces this further by enabling automatic model validation, attribute routing requirements, and application/json as the default content type.

Restart the app so it picks up the new controller and migration:

dotnet run --project src/Wassal.Monolith --launch-profile http

Leave it running. Open a second PowerShell window for the next task.

✅ Done when: the app starts on http://localhost:5011 and GET /api/orders returns [].

---

Task 3 — Demonstrate the BUG (then tag lesson-02-before) (5 min)

In your second PowerShell window, send the same POST twice — simulating a user double-click or an automatic retry:

$body = @{
  RestaurantId = 1
  Items        = "Koshary,Pizza"
  TotalAmount  = 95.50
} | ConvertTo-Json

$headers = @{ "Content-Type" = "application/json" }

# Same exact POST — twice
Invoke-RestMethod -Uri http://localhost:5011/api/orders -Method POST -Body $body -Headers $headers
Invoke-RestMethod -Uri http://localhost:5011/api/orders -Method POST -Body $body -Headers $headers

# Now count what's in the database
$orders = Invoke-RestMethod -Uri http://localhost:5011/api/orders
"$($orders.Count) order(s) in the database."
$orders | Format-Table id, restaurantId, totalAmount, createdAt

🐛 The bug: Two responses with DIFFERENT ids. Two rows in the database. You just charged the customer 95.50 × 2 = 191.00 for an order they placed once. In Wassal this is annoying. In a real payment system this is a refund ticket — or worse, fraud.

This is what production looks like without idempotency. Tag it as the BEFORE state and commit:

cd D:\books\distributed-system\Wassal
git add src/Wassal.Monolith/
git commit -m "feat: Lesson 2 — add naive POST /api/orders (idempotency-free, double-charge bug)"
git tag lesson-02-before

✅ Done when: the count says "2 order(s)" and tag lesson-02-before exists.

---

Task 4 — Apply the Fix — Idempotency-Key Handling (15 min)

The cure: when the client sends a unique Idempotency-Key header, the server remembers what it did for that key. A second request with the same key returns the same response instead of creating a second order.

We'll use an in-memory ConcurrentDictionary for the key→orderId map. This is the simplest possible implementation — perfect for learning, deliberately ignoring TTL, persistence, and multi-instance issues (which we'll address in later lessons).

Replace the contents of OrdersController.cs with this idempotent version:

// src/Wassal.Monolith/Controllers/OrdersController.cs (replace)
using System.Collections.Concurrent;
using Microsoft.AspNetCore.Mvc;
using Microsoft.EntityFrameworkCore;
using Wassal.Monolith.Data;
using Wassal.Monolith.Models;

namespace Wassal.Monolith.Controllers;

[ApiController]
[Route("api/[controller]")]
public class OrdersController : ControllerBase
{
    private readonly WassalDbContext _db;

    // In-memory store: idempotency key -> order id.
    // Static so it survives across requests (but lost on app restart — that's a known limitation
    // we'll address when we introduce Redis in a later lesson).
    private static readonly ConcurrentDictionary<string, int> _seenKeys = new();

    public OrdersController(WassalDbContext db) => _db = db;

    public record CreateOrderRequest(int RestaurantId, string Items, decimal TotalAmount);
    public record OrderResponse(int Id, int RestaurantId, decimal TotalAmount, bool Replayed);

    [HttpPost]
    public async Task<IActionResult> Create([FromBody] CreateOrderRequest req)
    {
        var key = Request.Headers["Idempotency-Key"].FirstOrDefault();

        // If the client provided a key we've seen before, replay the original response.
        if (key is not null && _seenKeys.TryGetValue(key, out var existingId))
        {
            var existing = await _db.Orders.FindAsync(existingId);
            if (existing is not null)
            {
                Response.Headers["Idempotency-Replayed"] = "true";
                return Ok(new OrderResponse(
                    existing.Id, existing.RestaurantId, existing.TotalAmount, Replayed: true));
            }
        }

        // Otherwise, create a new order.
        var order = new Order
        {
            RestaurantId = req.RestaurantId,
            Items        = req.Items,
            TotalAmount  = req.TotalAmount,
            CreatedAt    = DateTime.UtcNow,
        };
        _db.Orders.Add(order);
        await _db.SaveChangesAsync();

        // Remember the key → id mapping so future retries with the same key are deduped.
        if (key is not null)
            _seenKeys[key] = order.Id;

        return Ok(new OrderResponse(order.Id, order.RestaurantId, order.TotalAmount, Replayed: false));
    }

    [HttpGet]
    public async Task<IActionResult> List() =>
        Ok(await _db.Orders.OrderByDescending(o => o.CreatedAt).ToListAsync());
}

Stop the running app (Ctrl+C) and restart it so the new controller loads:

dotnet run --project src/Wassal.Monolith --launch-profile http

💡 Why in-memory? Because in this lesson we're teaching the shape of the solution, not production-grade infrastructure. Real systems put the key map in Redis (shared across instances) or a DB table (durable through restarts) with a TTL of a few hours. Same idea, different storage.

✅ Done when: the app rebuilds and starts on http://localhost:5011 with no errors.

---

Task 5 — Demonstrate the FIX (then tag lesson-02-after) (5 min)

This is the moment the lesson pays off. We're going to send three POST requests and watch the server treat them differently based on whether they share an idempotency key.

💡 What's about to happen — in plain words

1. Request #1 — we generate a fresh idempotency key (a random GUID like abc-123-…) and send the order. The server has never seen this key, so it creates Order #1 and records key → #1 in its in-memory ledger.
2. Request #2 — same body, SAME key (this simulates a retry — pretend the network ate the first response and the client tried again). The server checks its ledger, finds the key already there, and returns the cached Order #1 without writing to the database again.
3. Request #3 — same body, but a BRAND-NEW key (this represents a different intent — the user actually clicked "place another order"). The server has never seen this new key, so it creates Order #2.

Watch these three things in the output:

• Responses #1 and #2 have the same id, but #2 carries replayed: true
• Response #3 has a different id
• Final database count: 2 rows — one per unique intent, regardless of how many HTTP requests hit the server

The key insight: the server stopped counting "how many HTTP calls did I receive" and started counting "how many unique intents did the client express." Same intent (same key) = one row, no matter how many retries the network forces. New intent (new key) = new row, exactly as expected.

Step 0 — Clean slate. Clear yesterday's orders so the row count is unambiguous:

# Wipe the Orders table to start clean
sqlcmd -S "(local)" -E -d WassalDb -Q "DELETE FROM Orders;"

Steps 1 & 2 — Same key, twice (the retry scenario):

# ── THE PAYLOAD — same data for all three calls
$body = '{"restaurantId":1,"items":"Koshary,Pizza","totalAmount":95.50}'

# ── Generate ONE key. We'll re-use it for BOTH call #1 and call #2.
$key = [guid]::NewGuid().ToString()
"Generated Idempotency-Key: $key"

$headers = @{
  "Content-Type"    = "application/json"
  "Idempotency-Key" = $key
}

# Call #1 — server has never seen this key → creates Order #1
$response1 = Invoke-RestMethod -Uri "https://localhost:7126/api/orders" `
    -Method POST -Body $body -Headers $headers

# Call #2 — SAME key (retry) → server replays response #1, no new row
$response2 = Invoke-RestMethod -Uri "https://localhost:7126/api/orders" `
    -Method POST -Body $body -Headers $headers

"Response 1: id=$($response1.id)  replayed=$($response1.replayed)"
"Response 2: id=$($response2.id)  replayed=$($response2.replayed)"

# How many rows in the database now?
$orders = Invoke-RestMethod -Uri "https://localhost:7126/api/orders"
"$($orders.Count) order(s) — expected: 1"

💡 The fix in action: Both responses carry the same id, the second one says replayed: true, and only ONE row exists in the database. The customer was charged exactly once even though the request hit the server twice.

Step 3 — Different key, same body (a genuinely new intent — should create a new order):

# Fresh key = fresh intent. Server has never seen it.
$newKey = [guid]::NewGuid().ToString()
$headers["Idempotency-Key"] = $newKey
"New Idempotency-Key: $newKey"

$response3 = Invoke-RestMethod -Uri "https://localhost:7126/api/orders" `
    -Method POST -Body $body -Headers $headers
"Response 3: id=$($response3.id)  replayed=$($response3.replayed)"

# Now the count should be 2 — one row per UNIQUE intent
(Invoke-RestMethod -Uri "https://localhost:7126/api/orders").Count
# Expected: 2   (one from the duplicated key, one from the new key)

💡 Why [guid]::NewGuid()? A GUID is a 128-bit random ID with a vanishingly small chance of collision (you'd have to generate billions before any two matched). In a real frontend, you'd generate this once when the user clicks "Place Order" and re-use it for every retry of that one click. Different click → different GUID → different intent. It's how the client tells the server "this is a fresh thing I want done" vs "I'm just retrying my previous request."

Commit the AFTER state and tag it:

cd D:\books\distributed-system\Wassal
git add src/Wassal.Monolith/
git commit -m "fix: Lesson 2 — apply Idempotency-Key to POST /api/orders"
git tag lesson-02-after

✅ Done when: the duplicate POST shows replayed=true, the count is 1 (then 2 after the third unique-key request), and tag lesson-02-after exists.

---

Task 6 — Reflect, Document, Push (5 min)

Capture the before/after evidence in a short markdown file. Create it:

# Create the lessons folder
New-Item -ItemType Directory -Path docs\lessons -Force | Out-Null

# docs/lessons/lesson-02-idempotency.md
# Lesson 2 — REST + Idempotency

**Tags:** `lesson-02-before` → `lesson-02-after`
**Date:** 2026-05-20

## The pain (BEFORE)

POST /api/orders without idempotency. Two identical POSTs from the same client
produced two rows with different ids. The customer would be charged twice.

| Request | Response id | Rows in DB after |
|---------|-------------|-------------------|
| 1st     | 1           | 1                 |
| 2nd     | 2           | 2                 |

Root cause: HTTP itself has no way to distinguish "user retry" from "user re-intent".
The server has to be told.

## The fix (AFTER)

Client generates a unique `Idempotency-Key` per intent and re-uses it on retries.
Server keeps a `key -> orderId` map and replays the original response when it
sees a known key.

| Request                          | Response id | Replayed? | Rows in DB after |
|----------------------------------|-------------|-----------|-------------------|
| 1st  (key=X)                      | 5           | false     | 1                 |
| 2nd  (key=X, same)                | 5           | true      | 1                 |
| 3rd  (key=Y, new)                 | 6           | false     | 2                 |

## What's deliberately missing (will revisit later)

- Storage is `ConcurrentDictionary` — lost on app restart. Real systems use
  Redis or a DB table.
- No TTL. Keys live forever. Real systems expire them after 24h-7d.
- Single-instance only. With 3 app replicas, in-memory map isn't shared.
  We'll fix this when we introduce a Redis cache in a later lesson.
- The body isn't hashed. If the client sends DIFFERENT bodies with the SAME
  key, we silently return the first. Real systems either reject mismatches or
  hash the body into the key.

## Concept summary

Idempotency = same operation, same result, no matter how many times it's
performed. For HTTP: identical `POST` + same `Idempotency-Key` must produce
identical observable state changes regardless of how many times the network
delivers it.

Stage everything, commit, push:

git add docs/lessons/ docs/sessions/Day-03-Idempotency.html
git commit -m "docs: Lesson 2 writeup + Day 3 session plan"
git push origin master --tags

Verify all four lesson tags are now on the remote:

git tag
# expected:
#   lesson-01-before
#   lesson-01-pain-documented
#   lesson-02-before        ← new
#   lesson-02-after         ← new
#   v0-monolith-baseline

✅ Done when: 5 tags exist locally + on GitLab, and lesson-02-idempotency.md captures the before/after evidence.

---

📋 Quick Checklist

#	Task	Time	Done When
1	Order model + migration	10 min	Empty Orders table in WassalDb
2	Naive OrdersController	10 min	GET /api/orders returns []
3	Demo bug + tag lesson-02-before	5 min	Two rows from one intent
4	Add Idempotency-Key logic	15 min	App rebuilds with no errors
5	Demo fix + tag lesson-02-after	5 min	Same key → 1 row, new key → +1 row
6	Document, commit, push	5 min	5 tags on GitLab + lesson md

---

🎓 The pattern, repeated

Notice the shape of this lesson — it's the template every future Wassal lesson will follow: build the naive thing → show the pain → apply the concept → measure the fix → tag both. The git tags become a permanent before/after archive. Months from now you can git diff lesson-02-before lesson-02-after and see exactly what changed and why.

---

Part of the **Fundamentals of Distributed Systems* series — building Wassal, a distributed food-delivery lab, one concept at a time.*