DEV Community: mohamed Tayel

Day 16 — Retries, Backoff & Dead-Letter: Handling the Poison Message

mohamed Tayel — Sat, 25 Jul 2026 21:09:05 +0000

Day 15 decided who owns an outbox message. But leasing says nothing about what happens when processing keeps failing. A message that can never succeed — a poison message — will be retried forever unless retry has a limit, a backoff, and an exit.

🙏 Credit where it's due — This series is my attempt to internalize and share what I learned from Mahmoud Youssef's excellent course, Fundamentals of Distributed Systems on Udemy. The course material, structure, and topic flow are his work; the explanations, code examples, and diagrams in these articles are my own rewrite in my own words. If you find this useful, please consider taking the course — it goes much deeper than these articles can.

Where we are — after Days 13, 14 and 15

Day 13 — Durable Reconciliation: the pending work survives a restart.
Day 14 — Transactional Outbox: the payment row and the outbox row commit together, atomically.
Day 15 — Outbox Leasing: exactly one worker owns a message at a time, and ownership expires if the worker dies.
Day 16 (this document): leasing decides who processes a message — not what happens when processing keeps failing. A poison message needs bounded retries, backoff, and a dead-letter exit.

🎯 The one idea to take away: Leasing says who owns the work now. Retry says when to try again. Dead-letter says when to stop and ask a human. A durable, atomic, leased message can still fail every time it is processed. Without a limit and an exit, one bad message is retried forever and can starve the whole queue.

1. Why a leased, durable outbox can still loop forever

Everything so far assumed processing eventually succeeds. Real provider calls do not. They fail two ways:

Transient failure — a timeout, a blip, a temporary 503. Another try later will probably work.
Permanent failure — a malformed record, a rejected account, a bug. This message will never succeed. It is a poison message.
✅ Day 15 solved — ownership: one worker owns a message at a time; a dead owner's lease expires and another worker takes over.
❌ Day 16 solves — when to stop: a naive "retry until it works" loop never stops on a poison message — it re-claims and re-fails forever.

⚠️ The lease actually makes it worse. Because a message returns to the queue after each failure (or after its lease expires), the poison message is guaranteed to come back — forever — unless something counts the attempts and calls a halt.

2. The BEFORE scenario — retried forever, and the state hides no lie but never ends

session-16-poison-message-retried-forever seeds the Day 14 durable outbox (6 payments + 6 pending messages), then a single worker processes them with the most naive policy: on failure, put the message straight back to Pending and try again — no limit, no backoff, no exit. One message, PAY-003, is a poison message: its provider status check fails every time.

healthy ×5: PAY-001, 002, 004, 005, 006 each processed once → Processed (4 Paid, 1 Failed).
PAY-003 ×∞: claim → status check FAILS → back to Pending → claim → FAILS → … the attempt count climbs with no ceiling.
demo cap: a real run never ends, so the scenario stops after 10 retries and inspects the state: still Pending, AttemptCount = 10, never Processed, no dead-letter to catch it.

From the real run: healthy processed 5/5, poison retries 10, poison AttemptCount 10, poison final status Pending, provider status-check calls 15 (of which 10 failed), dead-lettered 0.

⚠️ Verdict from the real report: "A POISON MESSAGE RETRIED FOREVER — NO LIMIT, NO EXIT ❌". Nothing here is a lie or a double-charge — it is unbounded, wasted work that a shared queue would let starve every other message.

3. The retry columns and the new terminal state

The AFTER scenario adds a retry policy to the leased outbox row and one new status. A message now ends in one of two terminal states — Processed or DeadLettered:

Pending ─claim─▶ Processing ─success─▶ Processed        (terminal, good)
                     │
                     ├─fail, attempts < max ─▶ Pending + NextAttemptUtc  (backoff, retry later)
                     └─fail, attempts ≥ max ─▶ DeadLettered + LastError  (terminal, parked)

Column	Meaning
`AttemptCount` / `MaxAttempts`	How many times we've tried, and the ceiling before giving up.
`NextAttemptUtc`	Backoff gate — the message is not claimable again until this time.
`LastError`	Why the last attempt failed — recorded for the human who inspects a dead letter.

Status is now Pending | Processing | Processed | DeadLettered. The DeadLettered state is terminal but not lost — the message is parked, out of the worker's way, with its error attached.

4. The claim, now gated by backoff

The Day 15 atomic claim gets one extra clause: a message is only claimable if its backoff has elapsed. The claim stays a single BEGIN IMMEDIATE transaction with a conditional update:

BEGIN IMMEDIATE;
  SELECT Id FROM OutboxMessages
   WHERE Status IN ('Pending','Processing')
     AND (Status='Pending' OR LockedUntilUtc < $now)          -- lease (Day 15)
     AND (NextAttemptUtc IS NULL OR NextAttemptUtc <= $now)   -- backoff (Day 16)
   ORDER BY CreatedAtUtc LIMIT 1;
  UPDATE OutboxMessages SET Status='Processing', LockedBy=$worker,
         LockedUntilUtc=$lease, AttemptCount = AttemptCount + 1
   WHERE Id=$id AND … (same conditions) …;
COMMIT;

On failure the worker chooses one of two paths, purely from AttemptCount vs MaxAttempts:

if (AttemptCount < MaxAttempts)
    Status='Pending', NextAttemptUtc = now + backoff(AttemptCount), LastError=…   // retry later
else
    Status='DeadLettered', LastError=…                                            // give up, park it

5. Backoff — and proving it without waiting

Backoff spreads retries over time instead of hammering a struggling provider. Here the schedule is exponential: 200 ms × 2^(n-1) → 200 ms, then 400 ms. A message in backoff is genuinely not claimable — the worker's claim returns nothing, so it advances a deterministic ScenarioClock to the next attempt time (no real waiting). The real run's timeline for PAY-003:

t = 09:00:00.000Z: attempt 1/3 FAILED → backoff 200 ms → NextAttempt = 09:00:00.200Z (not claimable until then).
t = 09:00:00.201Z: attempt 2/3 FAILED → backoff 400 ms → NextAttempt = 09:00:00.601Z.
t = 09:00:00.602Z: attempt 3/3 FAILED → limit reached → DEAD-LETTERED (LastError recorded).

💡 Deterministic by design. Because the clock is injected, the backoff windows are exact and the test never sleeps. In production the same gate is a real wall-clock comparison; the logic is identical.

6. Dead-letter — parked, not lost

After MaxAttempts, the poison message moves to DeadLettered — a terminal state that takes it out of the worker's claim query so the queue keeps moving. Crucially, it is not deleted: its LastError is kept for a human or a redrive tool.

🔁 A retry loop says — never stop: keep trying the same failing work forever — burning provider calls and blocking the queue.
☠️ A dead-letter says — stop and surface: after N tries, park the message with its error so people can see it — and let everything else proceed.

💡 The payment is honestly still pending. Dead-lettering the message does not fake a payment outcome — PAY-003's payment row stays PendingConfirmation. We stopped the automatic retry; we did not invent a result.

7. The real implementation — proven by running it

Day 16 reuses the Day 14 outbox (OutboxSqliteCore) and the Day 15 lease model (Session15LeasingCore) unchanged, and adds a small retry core plus two scenarios registered in Program.cs:

BEFORE: Session16PoisonRetriedForeverScenario → command session-16-poison-message-retried-forever
AFTER: Session16RetryWithDeadLetterScenario → command session-16-retry-with-dead-letter

No new packages — the same Microsoft.Data.Sqlite. Standard .NET only: the atomic claim via BEGIN IMMEDIATE, a ScenarioClock for deterministic backoff windows, and a provider whose status check throws for the poison key (PAY-003). The poison message is a would-be Failed payment, so dead-lettering it loses no money.

Real BEFORE vs AFTER metrics

Metric (real run)	BEFORE (naive)	AFTER (policy)
Healthy messages Processed	5 / 5	5 / 5
Poison attempts	10 (capped; unbounded)	3 (= MaxAttempts)
Retries with backoff	0 (none)	2 (200 ms, 400 ms)
Poison final outbox status	Pending (never ends)	DeadLettered
Dead-lettered messages	0 (no such concept)	1 (LastError recorded)
Provider status-check calls	15 (10 failed)	8 (3 failed)
Outbox Pending / Processing at end	1 / 0 (stuck)	0 / 0 (drained)
Resolved Paid / Failed (healthy)	4 / 1	4 / 1
Duplicate payment / outbox rows	0 / 0	0 / 0
Fake successes	0	0

The headline: the poison message goes from an unbounded loop (10 and climbing, 10 wasted provider calls) to exactly 3 attempts and a clean exit — and the queue ends fully drained instead of stuck with one Pending row forever.

Safety checks from the real AFTER report — all passing

poison tried exactly MaxAttempts → 3 / 3
poison dead-lettered with LastError → status DeadLettered, error set
queue drained → 0 pending / 0 processing
no duplicate payment/outbox rows → max 1 per key
tx ids only from provider, Paid only → 4 Paid w/ id, 0 Failed w/ id

💡 Verdicts from the real reports. BEFORE: "A POISON MESSAGE RETRIED FOREVER — NO LIMIT, NO EXIT ❌". AFTER: "BOUNDED RETRY + DEAD-LETTER — THE QUEUE KEEPS MOVING ✅". Both exit with code 0 only when every check passes.

Run it yourself

dotnet run --project tools/Wassal.SessionTests -- session-16-poison-message-retried-forever
dotnet run --project tools/Wassal.SessionTests -- session-16-retry-with-dead-letter

Each run prints the schema, the processing/backoff timeline, the per-message table, the ledgers, the checks, and the verdict, then saves a timestamped report under:

Reports/Session-16/Session-16-Poison-Message-Retried-Forever-RunReport-<timestamp>.txt
Reports/Session-16/Session-16-Retry-With-Dead-Letter-RunReport-<timestamp>.txt

The SQLite file is created in a per-scenario temp folder — e.g. %LOCALAPPDATA%\Temp\wassal-day16-deadletter-<random>\wassal-outbox.db — and cleaned up at the end; its path is printed in the report.

8. Final takeaway

A durable, atomic, leased message can still fail every time — leasing decides ownership, not when to stop.
Retry needs a limit (MaxAttempts), a backoff (NextAttemptUtc gating re-claim), and an exit (DeadLettered).
A dead-lettered message is parked, not lost: terminal, out of the worker's way, with its LastError for a human.
Dead-lettering the message does not fake the payment — the payment stays honestly PendingConfirmation.
All Day 14/15 guarantees still hold: one payment per key, one outbox row per payment, exclusive ownership, no duplicate charges, provider-only transaction ids.

The four days, side by side:

Day 13:  The work survives a restart.
Day 14:  The business state and outbox row commit together.
Day 15:  Only one worker owns a message at a time.
Day 16:  A message that always fails is tried a bounded number of times, then parked.

💡 Leasing says who owns the work now. Retry says when to try again. Dead-letter says when to stop and ask a human. Retry scheduling refinements, a hosted worker, and dead-letter redrive tooling are later lessons.

Part of the **Fundamentals of Distributed Systems* series — building Wassal, a distributed food-delivery lab, one concept at a time.*

Day 15 — Outbox Leasing: Preventing Competing Workers from Processing the Same Message

mohamed Tayel — Sat, 25 Jul 2026 21:08:27 +0000

Day 14 made the payment and its outbox message commit atomically, and Day 13 made them durable. But a durable message with two workers is a shared message — both can read it and process it. Day 15 gives each message one temporary owner: a lease.

🙏 Credit where it's due — This series is my attempt to internalize and share what I learned from Mahmoud Youssef's excellent course, Fundamentals of Distributed Systems on Udemy. The course material, structure, and topic flow are his work; the explanations, code examples, and diagrams in these articles are my own rewrite in my own words. If you find this useful, please consider taking the course — it goes much deeper than these articles can.

Where we are — after Days 13 and 14

Day 13 — Durable Reconciliation: the pending work survives a restart — a fresh instance reloads it from durable storage. The work survives a restart.
Day 14 — Transactional Outbox: the payment row and the outbox row commit or roll back together in one SQLite transaction. The business state and outbox row commit together.
Day 15 (this document): in a real deployment there is more than one application instance, each running an outbox worker. Two workers can read the same pending row. Only one worker owns a message at a time — and ownership expires if the worker dies.

🎯 The one idea to take away: Durability preserves the work. Atomicity creates the work safely. Leasing decides who owns the work now. The message is durable — now we need exclusive, temporary ownership. Durable does not mean exclusively owned; pending does not mean safe for every worker to process.

1. Why a durable, atomic outbox can still be processed twice

Day 14 answered "is the message durable and consistent?" with a firm yes. But that is a different question from "who is allowed to process it right now?"

✅ Days 13–14 solved — durability + atomicity: the message survives restart and always commits with its payment. One row, no orphans.
❌ Day 15 solves — exclusive ownership: with two app instances, both workers SELECT … WHERE Status='Pending' and both process the same rows.

A single-worker deployment hides this. Scale to two instances and the naive worker loop becomes a race: Worker A reads the pending rows, Worker B reads the same pending rows, and both start processing before either marks anything done.

⚠️ For payment reconciliation the work is a status read, so a duplicate does not double-charge. But it is duplicate provider load, wasted work, competing DB updates, and misleading metrics — and for other message types (send an email, ship an order) a duplicate is a real, visible side effect.

2. The BEFORE scenario — competing workers, and why the final state hides it

session-15-competing-workers-no-claim seeds the Day 14 durable outbox (6 payments + 6 pending messages), then starts two workers, each on its own SQLite connection (separate application instances). A Barrier(2) makes the race deterministic — neither worker processes until both have loaded the full pending snapshot:

Load: Worker-A loads all 6 pending rows. Worker-B loads all 6 pending rows. Neither has marked anything processed yet.
Process: both workers call CheckPaymentStatus for every message they loaded — 12 processing attempts, 12 provider status checks for 6 unique messages.
Final DB: all 6 rows end Processed, 4 Paid + 2 Failed, no duplicate rows. The database looks perfect.

That tidy final state is the trap. From the real run: unique messages 6, Worker-A loaded 6, Worker-B loaded 6, processing attempts 12, status checks 12, messages processed more than once 6. The doubling is only visible because the report keeps the ledgers separate.

⚠️ Verdict from the real report: "THE OUTBOX WAS DURABLE — BUT TWO WORKERS PROCESSED THE SAME WORK ❌". Final row state alone cannot tell you whether work happened once or twice.

3. The lease columns and message states

The AFTER scenario adds two columns and a third status. A message now moves through:

Pending  ──claim──▶  Processing  ──finalize──▶  Processed
                         │
                         └── lease expires ──▶ claimable again

New column	Meaning
`LockedBy`	The worker id that currently owns the message (NULL when not leased).
`LockedUntilUtc`	When the lease expires. After this time another worker may reclaim the message.

Status is now Pending | Processing | Processed. A message is claimable when it is Pending, or Processing with an expired lease.

4. The atomic claim — and claim vs processing

The claim must be atomic, or the race just moves from "process" to "claim". A worker claims inside one BEGIN IMMEDIATE transaction (SQLite takes the write lock up front, serializing claimers) with a conditional update:

BEGIN IMMEDIATE;
  SELECT Id FROM OutboxMessages
   WHERE Status='Pending' OR (Status='Processing' AND LockedUntilUtc < $now)
   ORDER BY CreatedAtUtc LIMIT 1;
  UPDATE OutboxMessages
     SET Status='Processing', LockedBy=$worker, LockedUntilUtc=$leaseUntil
   WHERE Id=$id
     AND (Status='Pending' OR (Status='Processing' AND LockedUntilUtc < $now));
COMMIT;   -- claimed only if this UPDATE affected exactly one row

The two operations are deliberately different, and the implementation keeps two separate ledgers for them:

Claim = acquire temporary ownership of a row. Cheap, atomic, may fail (someone else won).
Processing = the actual work (CheckPaymentStatus, apply outcome). Happens only for messages you successfully claimed.

💡 Claim attempts ≠ work done. In the real AFTER run there were 5 successful claims in the concurrent phase plus a handful of empty claims (nothing claimable at that instant) — but exactly 6 unique messages were processed, once each.

5. Why leases expire — a lease is not a lock

A permanent lock would be a bug waiting to happen: if the owner crashes mid-work, the message is locked forever and the work is lost. A lease is ownership with an expiry.

🔒 A lock says — permanent: "Nobody else may ever take this work." If the owner dies, the work is stuck forever.
🎟️ A lease says — temporary: "I own this work until a specific time. If I disappear, another worker may recover it after expiry."

⚠️ Why the lease matters. A worker may claim a message and then crash. A permanent lock would lose the work forever. An expired lease lets another worker recover it — which is exactly what the recovery sub-scenario below proves.

6. Worker crash after claim, then reclaim by another worker

The AFTER scenario includes a small deterministic recovery demo on a scenario-controlled clock (so lease expiry needs no real waiting). Real run, verbatim from the report:

t = 09:00:00.000Z: Worker-A claims PAY-001 → CLAIMED (LockedBy=Worker-A, lease 500 ms).
crash: Worker-A crashes before processing — the row stays Status=Processing, LockedBy=Worker-A.
before expiry: Worker-B tries to claim PAY-001 → BLOCKED (owner still Worker-A, lease not expired).
t = 09:00:00.600Z: advance the clock past the lease → Worker-B RECLAIMS PAY-001 (LockedBy=Worker-B).
process: Worker-B processes PAY-001 exactly once → outbox=Processed. The work was never lost.

A lock would have left PAY-001 stuck under a dead owner. The lease let a healthy worker take over — safely, and only after the previous owner's claim demonstrably expired.

7. Ownership-verified finalization

Claiming is only half the guarantee — completing must also check ownership, so a worker never finalizes a message it no longer owns (e.g. one whose lease expired and was reclaimed). Payment finalization and message completion happen in one transaction, guarded by LockedBy:

BEGIN IMMEDIATE;
  UPDATE Payments SET PaymentStatus=…, TransactionId=… WHERE IdempotencyKey=$key;
  UPDATE OutboxMessages
     SET Status='Processed', ProcessedAtUtc=$now, LockedUntilUtc=NULL
   WHERE Id=$id AND Status='Processing' AND LockedBy=$worker;   -- ownership check
COMMIT;   -- only if that UPDATE affected exactly one row; otherwise ROLLBACK

💡 Ownership violations = 0 in the real run. If the guarded update affects zero rows, the worker does not own the message any more and the transaction rolls back — it never writes a result it has no right to write.

8. The real implementation — proven by running it

Day 15 reuses the Day 14 SQLite infrastructure (OutboxSqliteCore) unchanged and adds a small leasing core plus two scenarios registered in Program.cs:

BEFORE: Session15CompetingWorkersNoClaimScenario → command session-15-competing-workers-no-claim
AFTER: Session15OutboxLeasingScenario → command session-15-outbox-leasing

No new packages — the same Microsoft.Data.Sqlite. Standard .NET only: a Barrier(2) to make the BEFORE race deterministic, two SqliteConnections (WAL + busy_timeout) for two application instances, BeginTransaction(deferred:false) for BEGIN IMMEDIATE atomic claims, and a small ScenarioClock so lease expiry is deterministic without real waiting.

Real BEFORE vs AFTER metrics

Metric (real run)	BEFORE (no claim)	AFTER (leasing)
Payments / pending outbox seeded	6 / 6	6 / 6
Worker-A loaded / Worker-B loaded	6 / 6 (both, all rows)	claimed per-message, not bulk-loaded
Processing attempts	12	6
Unique messages processed	6	6
Messages processed more than once	6	0
Provider status-check calls	12	6
Ownership violations	n/a (no ownership)	0
Worker ownership split (A / B)	both did all 6	3 / 3 (varies by who wins each claim)
Resolved Paid / Failed / pending	4 / 2 / 0	4 / 2 / 0
Outbox Processed / Pending / Processing	6 / 0 / 0	6 / 0 / 0
Duplicate payment / outbox rows	0 / 0	0 / 0
Payment creation calls in workers	0	0
Fake successes	0	0

The headline is the halving: 12 → 6 processing attempts and status checks for the same 6 messages. Same durable outcome, half the provider load, zero duplicate work. The A/B split is whatever the atomic claims happen to produce (3/3 in this run; 2/4 in another) — the invariant is 6 unique, once each.

Lease-expiry recovery proof (AFTER)

From the real report: Worker-A claimed PAY-001 then crashed → Worker-B blocked before expiry (true) → advance clock → Worker-B reclaimed after expiry (true) → reclaimed message processed once. A lock would have lost PAY-001; the lease recovered it.

Safety checks from the real AFTER report — all passing

each message processed exactly once → 6 unique, 0 duplicates
zero ownership violations → 0
no duplicate payment/outbox rows → max 1 per key
one gateway creation attempt per key → 6 / 6
tx ids only from provider, Paid only → 4 Paid w/ id, 0 Failed w/ id

💡 Verdicts from the real reports. BEFORE: "THE OUTBOX WAS DURABLE — BUT TWO WORKERS PROCESSED THE SAME WORK ❌". AFTER: "OUTBOX LEASING — ONE OWNER PER MESSAGE, AND OWNERSHIP EXPIRES ✅". Both exit with code 0 only when every check passes.

Run it yourself

dotnet run --project tools/Wassal.SessionTests -- session-15-competing-workers-no-claim
dotnet run --project tools/Wassal.SessionTests -- session-15-outbox-leasing

Each run prints the schema, the worker/claim timeline, the lease-expiry demo, the per-message ownership table, the ledgers, the checks, and the verdict, then saves a timestamped report under:

Reports/Session-15/Session-15-Competing-Workers-No-Claim-RunReport-<timestamp>.txt
Reports/Session-15/Session-15-Outbox-Leasing-RunReport-<timestamp>.txt

The SQLite file is created in a per-scenario temp folder — e.g. %LOCALAPPDATA%\Temp\wassal-day15-leasing-<random>\wassal-outbox.db — and cleaned up at the end; its path is printed in the report.

9. Final takeaway

A durable, atomic outbox is still a shared outbox — two workers can read and process the same pending row.
Final row state cannot prove single processing; you must count claim and processing attempts separately.
An atomic claim (BEGIN IMMEDIATE + conditional update) gives exactly one worker temporary ownership of a message.
Finalization must verify ownership (LockedBy = me) so a worker never completes a message it no longer owns.
Ownership is a lease, not a lock: it expires, so a crashed owner's work can be reclaimed instead of lost forever.
All Day 14 guarantees still hold: one payment per key, one outbox row per payment, no orphans, no duplicate charges, provider-only transaction ids.

The three days, side by side:

Day 13:  The work survives a restart.
Day 14:  The business state and outbox row commit together.
Day 15:  Only one worker owns a message at a time — and ownership expires if the worker dies.

💡 Durability preserves the work. Atomicity creates it safely. Leasing decides who owns it now. Retry backoff, dead-letter handling, and a hosted worker are later lessons — Day 15's invariant is exactly one temporary owner per message.

Part of the **Fundamentals of Distributed Systems* series — building Wassal, a distributed food-delivery lab, one concept at a time.*

Day 14 — Transactional Outbox: Saving the Payment and the Promise Atomically

mohamed Tayel — Sat, 25 Jul 2026 21:08:21 +0000

Day 13 made the follow-up work survive a restart — but it saved the payment and the promise as one JSON file, deliberately labelled outbox-shaped. Day 14 moves to a real SQLite database and asks the sharper question: can the payment be saved while its work item goes missing? The answer is a single database transaction.

🙏 Credit where it's due — This series is my attempt to internalize and share what I learned from Mahmoud Youssef's excellent course, Fundamentals of Distributed Systems on Udemy. The course material, structure, and topic flow are his work; the explanations, code examples, and diagrams in these articles are my own rewrite in my own words. If you find this useful, please consider taking the course — it goes much deeper than these articles can.

🧭 Where we are — after Days 11, 12 and 13

Day 11 — Honest Fallback: on a payment timeout, return PendingConfirmation honestly. We admitted the result was unknown.
Day 12 — Reconciliation: a worker calls CheckPaymentStatus(originalKey) and resolves each pending payment. We learned how to check the original attempt.
Day 13 — Durable Reconciliation: the pending work is persisted so it survives a restart — as one JSON snapshot, outbox-shaped, not transactional. We made the follow-up work survive a restart.
Day 14 (this document): the payment record and its reconciliation obligation must be committed together. We commit the business state and the follow-up work in one database transaction.

🎯 The one idea to take away: Durability prevents forgetting after restart. Atomicity prevents forgetting during the write. Save the business state + save the follow-up work = one database transaction. Either both commit, or neither does.

1. Why Day 13's JSON store was outbox-shaped, not transactional

Day 13 solved a real problem — in-memory work vanishes on restart, so we wrote it to disk with a safe atomic file replace. That answered one question well:

✅ Day 13 solved — durability: Will the saved work survive a restart? Yes — a fresh instance reloads the pending work from disk.

❌ Day 14 solves — atomicity: Can the payment be saved while the work item is missing? In a database with two separate writes — yes, and that is the bug.

Day 13's snapshot always wrote the payment and the work item in the same file, so they never diverged — but that convenience hid the real production hazard. In a database-backed system the natural implementation is two writes: save the payment, then save the outbox row. That is the dual write, and it has a crash window.

💡 Outbox-shaped ≠ transactional. A single JSON document with an atomic replace is not a database transaction spanning a business row and an outbox row. Day 14 builds the real thing on SQLite so the transaction boundary is visible.

2. The dual-write problem — a crash window between two durable writes

The dangerous pattern is two independent, separately-committed writes:

INSERT Payment   → COMMIT      // durable
   ... crash ...               // the window
INSERT Outbox    → COMMIT      // never happens

If the process dies in the window, the database is left durably inconsistent:

Payment exists:   PAY-002 = PendingConfirmation
Outbox row:       missing

The payment is perfectly durable — and completely forgotten. No worker has a row telling it this payment needs reconciling, so it stays pending forever even after the provider recovers holding the real answer. The mirror-image inconsistency (outbox row written first, payment missing) is equally possible if you flip the order.

💡 Durability did not save us. Both rows were individually durable. The missing guarantee is that they become durable together — all or nothing.

3. The SQLite schema

Both scenarios use the same small, explicit schema in a real local SQLite file (Microsoft.Data.Sqlite, direct SQL — no EF Core, no server):

CREATE TABLE Payments (
    IdempotencyKey       TEXT PRIMARY KEY,       -- the business identity
    PaymentStatus        TEXT NOT NULL,
    PaymentConfirmed     INTEGER NOT NULL,
    RequiresStatusCheck  INTEGER NOT NULL,
    IsFallback           INTEGER NOT NULL,
    TransactionId        TEXT NULL,
    CreatedAtUtc         TEXT NOT NULL,
    UpdatedAtUtc         TEXT NOT NULL
);

CREATE TABLE OutboxMessages (
    Id                   TEXT PRIMARY KEY,        -- deterministic per payment
    MessageType          TEXT NOT NULL,
    AggregateId          TEXT NOT NULL,           -- FK → Payments(IdempotencyKey)
    Payload              TEXT NOT NULL,
    Status               TEXT NOT NULL,           -- Pending | Processed
    AttemptCount         INTEGER NOT NULL,
    CreatedAtUtc         TEXT NOT NULL,
    ProcessedAtUtc       TEXT NULL,
    FOREIGN KEY (AggregateId) REFERENCES Payments(IdempotencyKey)
);

CREATE INDEX IX_Outbox_Pending ON OutboxMessages(Status) WHERE Status = 'Pending';

The Payments.IdempotencyKey primary key is the DB-level guard against a duplicate payment; each outbox row uses a deterministic id such as OUTBOX-PAYMENT-RECONCILIATION-PAY-001, so a retried write can never create a duplicate work item either.

4. The BEFORE scenario — the dual-write gap, proven

session-14-dual-write-gap replays Days 11–13 (6 payments, hung gateway, 300 ms timeout, SemaphoreSlim(2) → all PendingConfirmation), then persists each with the dual write. A controlled failure fires between the two writes for PAY-002, PAY-004, PAY-006:

PAY-001/003/005: payment=COMMIT then outbox=COMMIT — both rows land.
PAY-002/004/006: payment=COMMIT → injected crash → outbox=MISSING. The payment is durable; its obligation never gets written.
Restart: a fresh Instance 2 opens the same DB and reads only the 3 pending outbox rows it can find. It reconciles those 3; the other 3 stay pending forever.

Real run: payments persisted 6, outbox persisted 3, payments missing outbox 3, orphan outbox 0. The 3 reconciled split into 1 Paid (PAY-001) + 2 Failed (PAY-003, PAY-005), leaving 3 still pending (PAY-002/004/006 — which the provider actually charged).

💡 Verdict from the real report: "THE DATA WAS DURABLE — BUT THE FOLLOW-UP WAS NOT ATOMIC ❌". Three customers were charged, and nothing in our system will ever ask about them.

5. The AFTER fix — one transaction, commit or rollback

session-14-transactional-outbox writes both rows on one connection inside one SqliteTransaction:

BEGIN TRANSACTION
  INSERT Payment       ON CONFLICT(IdempotencyKey) DO NOTHING
  INSERT OutboxMessage ON CONFLICT(Id)             DO NOTHING
COMMIT           -- both rows land together
      // or, on failure before COMMIT:
ROLLBACK         -- neither row remains

The same failure is injected inside the transaction, before COMMIT. From the real write timeline:

PAY-001 | BEGIN → payment+outbox → COMMIT
PAY-002 | BEGIN → payment → ROLLBACK   ← injected failure
PAY-002 | BEGIN → payment+outbox → COMMIT (retry #2)
PAY-003 | BEGIN → payment+outbox → COMMIT
PAY-004 | BEGIN → payment → ROLLBACK   ← injected failure
PAY-004 | BEGIN → payment+outbox → COMMIT (retry #2)
PAY-005 | BEGIN → payment+outbox → COMMIT
PAY-006 | BEGIN → payment → ROLLBACK   ← injected failure
PAY-006 | BEGIN → payment+outbox → COMMIT (retry #2)

3 rollbacks, 6 commits. After every rollback the operation is retried with the same idempotency key and commits both rows cleanly. The rollback is not hidden — it is in the report, on purpose.

💡 Commit-or-rollback is the whole point. A rolled-back transaction leaves nothing — no half-written payment, no orphan outbox row. There is never a moment where one row exists without the other.

6. Consistency rules and the role of the idempotency key

Retrying after a rollback must not create duplicates. The safety comes from the database, not an in-memory check:

Same IdempotencyKey = same payment operation. It is the primary key, so a second committed payment for the same key is impossible.
Deterministic outbox id (OUTBOX-PAYMENT-RECONCILIATION-<key>) is the outbox primary key, so a duplicate work row is impossible.
Both inserts use ON CONFLICT DO NOTHING, so a retry is a safe no-op if a row already exists.

The two durable invariants the AFTER report verifies with SQL, not assertions:

SELECT COUNT(*) FROM Payments p
 WHERE NOT EXISTS (SELECT 1 FROM OutboxMessages o WHERE o.AggregateId = p.IdempotencyKey);  -- = 0

SELECT COUNT(*) FROM OutboxMessages o
 WHERE NOT EXISTS (SELECT 1 FROM Payments p WHERE p.IdempotencyKey = o.AggregateId);        -- = 0

💡 Never a payment without outbox; never an outbox without payment. Real run: payments missing outbox = 0, outbox missing payments = 0, duplicate payment rows = 0, duplicate outbox rows = 0.

7. Restart and outbox processing — check, never charge

After the writes, a simulated crash discards Instance 1; a fresh Instance 2 opens the same SQLite file and processes the pending outbox:

SELECT Id, AggregateId FROM OutboxMessages
 WHERE Status = 'Pending' ORDER BY CreatedAtUtc;

For each pending message, inside its own transaction, the worker calls CheckPaymentStatus(originalKey) — a status read, never CreatePayment — applies the provider's real result, and marks the message Processed. Finalizing the payment and completing the message commit together, so a payment can never be finalized while its message stays pending:

PAY-001 | PendingConfirmation → Paid   | outbox Pending → Processed | TX-PAY-001
PAY-002 | PendingConfirmation → Paid   | outbox Pending → Processed | TX-PAY-002
PAY-003 | PendingConfirmation → Failed | outbox Pending → Processed | TransactionId=null
PAY-004 | PendingConfirmation → Paid   | outbox Pending → Processed | TX-PAY-004
PAY-005 | PendingConfirmation → Failed | outbox Pending → Processed | TransactionId=null
PAY-006 | PendingConfirmation → Paid   | outbox Pending → Processed | TX-PAY-006

Our application restarted. The SQLite database survived. The external payment provider also survived. The provider is a separate simulated system that keeps its original outcomes across our restart; the SQLite file is our durable state; no static field carries business data between the two instances.

💡 Creation vs check — still separate ledgers. 6 gateway creation attempts (Phase 1, one per key), 0 creation calls during the worker, 6 status checks after restart. Reconciliation reads the original attempt; it never starts a new one.

8. The real implementation — proven by running it

Day 14 is three files under the session-test project: a shared OutboxSqliteCore (schema, provider, consistency queries, the shared Phase 1) and two ISessionScenario classes registered in Program.cs:

BEFORE: Session14DualWriteGapScenario → command session-14-dual-write-gap
AFTER: Session14TransactionalOutboxScenario → command session-14-transactional-outbox

The only new dependency is Microsoft.Data.Sqlite. Everything else is standard .NET — Task.Delay for the hung gateway and the status read, SemaphoreSlim(2) for the bulkhead, a linked CancellationTokenSource for the 300 ms timeout, and direct SQL over a real SQLite file created in a per-scenario temp folder and deleted afterwards.

Real BEFORE vs AFTER metrics

Metric (real run)	BEFORE (dual write)	AFTER (one transaction)
Write-transaction commits	9 (6 payment + 3 outbox, separate)	6 (payment + outbox together)
Write-transaction rollbacks	0 (nothing to roll back)	3 (then retried)
Committed payment rows	6	6
Committed outbox rows	3	6
Payments missing outbox rows	3	0
Outbox rows missing payments	0	0
Pending outbox loaded on restart	3	6
Status checks after restart	3	6 (one per key)
New payment attempts during worker	0	0
Resolved as Paid	1	4
Resolved as Failed	2	2
Still pending	3	0
Processed outbox rows	3	6
Duplicate payment / outbox rows	0 / 0	0 / 0
Fake successes	0	0

Transaction commit & rollback evidence (AFTER)

The real write timeline shows all three rollbacks and their retries: PAY-002, PAY-004 and PAY-006 each go BEGIN → payment → ROLLBACK, then BEGIN → payment+outbox → COMMIT (retry #2). Final: 6 write commits, 3 rollbacks, 6 processing commits.

Safety checks from the real AFTER report — all passing

no duplicate payment rows → max 1 row per key
no duplicate outbox rows → max 1 row per key
one gateway creation attempt per key → 6 / 6
zero fake successes → 0
tx ids only from provider, Paid only → 4 Paid w/ id, 0 Failed w/ id

💡 Verdicts from the real reports. BEFORE: "THE DATA WAS DURABLE — BUT THE FOLLOW-UP WAS NOT ATOMIC ❌". AFTER: "TRANSACTIONAL OUTBOX — THE PAYMENT AND THE PROMISE COMMITTED TOGETHER ✅". Both exit with code 0 only when every check passes.

Run it yourself

dotnet run --project tools/Wassal.SessionTests -- session-14-dual-write-gap
dotnet run --project tools/Wassal.SessionTests -- session-14-transactional-outbox

Each run prints the schema, the write timeline, the consistency query results, the per-payment table, the summary, the checks, and the verdict, then saves a timestamped report under:

Reports/Session-14/Session-14-Dual-Write-Gap-RunReport-<timestamp>.txt
Reports/Session-14/Session-14-Transactional-Outbox-RunReport-<timestamp>.txt

The SQLite file is created in a per-scenario temp folder — e.g. %LOCALAPPDATA%\Temp\wassal-day14-outbox-<random>\wassal-outbox.db — and cleaned up at the end; its path is printed in the report.

9. Final takeaway

Two separate durable writes have a crash window that can leave a payment without its follow-up obligation — durability alone does not close it.
Writing the payment row and the outbox row in one database transaction makes them commit or roll back together.
A rollback leaves nothing; a retry with the same idempotency key is safe because DB primary keys enforce one payment and one outbox row per key.
On restart, the worker reads pending outbox rows and checks the original attempt — it never re-charges — finalizing payment and message in one transaction.
This is now a real local transactional outbox demonstration using SQLite — not outbox-shaped. Retries with backoff, leasing, dead-letter handling, and a hosted worker are later lessons.

The four days, side by side:

Day 11:  We admitted that the result was unknown.
Day 12:  We learned how to check the original attempt.
Day 13:  We made the follow-up work survive a restart.
Day 14:  We commit the payment state and the follow-up obligation together.

💡 Day 13 solved: will the saved work survive restart? Day 14 solves: can the payment be saved while the work item is missing? Durability prevents forgetting after restart; atomicity prevents forgetting during the write.

Six payments, three injected failures, three rollbacks, six clean commits — then a restart that reloaded all six outbox rows and resolved them: 4 Paid, 2 Failed, 0 pending, 0 duplicates, 0 fake successes. Save the business state and the follow-up work in one database transaction: both, or neither.

Part of the **Fundamentals of Distributed Systems* series — building Wassal, a distributed food-delivery lab, one concept at a time.*

Day 13 — Durable Reconciliation: Surviving Application Restarts

mohamed Tayel — Sat, 25 Jul 2026 21:07:41 +0000

Day 12 built a worker that resolves pending payments by asking the provider — but the pending work lived only in memory. Day 13 asks the harder question: what happens when the process crashes before that work runs? In-memory work disappears. Durable work survives.

🙏 Credit where it's due — This series is my attempt to internalize and share what I learned from Mahmoud Youssef's excellent course, Fundamentals of Distributed Systems on Udemy. The course material, structure, and topic flow are his work; the explanations, code examples, and diagrams in these articles are my own rewrite in my own words. If you find this useful, please consider taking the course — it goes much deeper than these articles can.

🧭 Where we are — after Days 11 and 12

Day 11 — Payment Fallback Without Lying: when the provider times out, we return an honest degraded response — PendingConfirmation, PaymentConfirmed = false, RequiresStatusCheck = true, idempotency key preserved, no fabricated transaction id. "We do not know yet."
Day 12 — Payment Reconciliation: a worker scans the pending store and calls CheckPaymentStatus(originalKey) — a status read, never a re-charge — and moves each payment to the provider's real final state. "We checked the original attempt and learned the result."
Day 13 (this document): in Day 12 the pending store and the reconciliation queue lived in memory, inside one process. If that process crashes before the worker runs, the work is gone. Even if our application crashes, it must remember that the check is still required.

🎯 The one idea to take away: Pending is a promise. Durability makes sure the system remembers the promise. In-memory work disappears when the process dies. Durable work survives restarts. Knowing what to do is not enough — the work must outlive the process that created it.

1. Why in-memory background work is unsafe

Day 12's worker was correct — but it was only as durable as the process it ran in. A reconciliation queue held in a List<WorkItem> and a pending store held in a Dictionary share one fate: the moment the process exits, they are gone. Real processes exit all the time — a deploy, an OOM kill, a crash, a pod eviction, a machine reboot.

The dangerous window is small but real:

A payment times out and becomes PendingConfirmation, RequiresStatusCheck = true.
A reconciliation work item is enqueued in memory.
The application crashes before the worker runs.
The in-memory pending work disappears.
After restart, the application does not remember what it must check.

✅ The worker (Day 12) — solved: knows exactly how to resolve a pending payment: CheckPaymentStatus(key), never re-charge.

❌ Remembering the work (Day 13 · still missing): the queue lives in memory. A restart erases it, and the recovered provider's answers are never collected.

💡 The failure mode here is not a double charge — it is forgetting. The provider may recover holding the real outcome, but our side no longer has a queued item asking about it. The follow-up obligation vanished with the process that created it.

2. Our application state vs the external provider's state

This distinction is the heart of the lesson. Two systems, two separate fates when our process crashes:

🧠 Our application state — volatile: pending store + reconciliation queue. Held in memory → lost on crash unless it is made durable.

🏦 The external provider — survives: a separate system. It never lost its records — it can be hung, then recover, still holding every original outcome.

Our application restarted.
The external payment provider did NOT lose its records.

In both scenarios the provider is modelled as its own object (ExternalPaymentProvider) created once and handed to both application instances — because a real provider does not restart when our process does. The gap Day 13 closes is entirely on our side: remembering that a check is still owed.

3. The BEFORE problem — the crash erases the work

The BEFORE scenario (session-13-in-memory-work-lost) replays Day 11/12, then models a real process boundary:

Phase 1 — Application Instance 1. 6 payments (PAY-001…PAY-006) hit a hung gateway (2000 ms) behind the SemaphoreSlim(2) bulkhead with the 300 ms timeout. All 6 time out → 6 PendingConfirmation records and 6 reconciliation work items, all in memory only.
Simulated crash — Instance 1 is discarded. Every reference to its pending dictionary and its reconciliation queue is dropped and a brand-new Instance 2 is constructed — no shared references, no static state. This is what a crash does to memory, not a list.Clear().
Phase 3 — the provider recovers, still holding every answer (4 charged, 2 failed). But Instance 2's queue is empty. Work items after restart: 0. Status checks: 0. Resolved: 0. The obligation is gone.

From the real run: pending before crash 6, in-memory work items before crash 6, work items after restart 0, status checks after restart 0, resolved payments 0, still unresolved externally 6, duplicate charge attempts 0.

💡 Verdict from the real report: "THE WORKER KNEW WHAT TO DO — BUT THE CRASH ERASED THE WORK ❌". Every check in the BEFORE report passes — because the problem it proves is real: a correct worker with no durable memory forgets its obligations the instant the process dies.

4. The AFTER fix — a durable reconciliation store

The AFTER scenario (session-13-durable-reconciliation) changes one thing: where the pending work lives. When a payment becomes PendingConfirmation, its payment record and its reconciliation work item are persisted together as one JSON snapshot, written with a safe pattern:

write temporary file
flush / close   (fsync — force the bytes to disk)
atomically move the temp file over the final file

Instance 1 creates the payment attempts and persists each pending payment + work item durably.
Simulated crash: all of Instance 1's in-memory state is discarded.
Instance 2 starts with empty memory, reloads the six pending work items only from durable storage, and reconciles using status checks only.

From the real run, every payment reloaded and resolved to the provider's truth:

PAY-001 | PendingConfirmation → Paid    | TX-PAY-001
PAY-002 | PendingConfirmation → Paid    | TX-PAY-002
PAY-003 | PendingConfirmation → Failed  | TransactionId=null
PAY-004 | PendingConfirmation → Paid    | TX-PAY-004
PAY-005 | PendingConfirmation → Failed  | TransactionId=null
PAY-006 | PendingConfirmation → Paid    | TX-PAY-006

💡 This is outbox-shaped, not a production Transactional Outbox. A single JSON file plus an atomic replace buys us restart-survival — but it is not a database transaction that commits the business write and the outbox row together. A real transactional outbox, a hosted background worker, schedules, retries, and dead-letter handling are later lessons. Day 13 introduces the direction, honestly labelled.

5. The durable work-item model

One durable snapshot holds two lists — payments and their work items — persisted together so a restart never sees a payment without its follow-up obligation (or vice-versa):

Durable payment record	Durable reconciliation work item
`IdempotencyKey`	`WorkItemId`
`PaymentStatus`	`PaymentIdempotencyKey`
`PaymentConfirmed`	`Status` — Pending \| Completed
`RequiresStatusCheck`	`AttemptCount`
`TransactionId`	`CreatedAt`
`IsFallback`	`CompletedAt` (when completed)

💡 Persisted together, as one logical snapshot. The payment state and its work item are written in a single atomic replace — so the durable store never records a pending payment whose reconciliation obligation was lost, which is exactly the failure the BEFORE scenario proved.

6. State transitions — before and after the restart

Every transition is driven by the provider's reported truth — reloaded from disk, never guessed:

Before crash:  PendingConfirmation | WorkItem = Pending   (durably saved)
After restart: Loaded from durable store                  (in-memory memory was empty)
After check:   Paid or Failed        | WorkItem = Completed (durably updated)

Final state	PaymentConfirmed	RequiresStatusCheck	IsFallback	TransactionId	WorkItem
`Paid`	true	false	false	real provider id (e.g. TX-PAY-001)	Completed
`Failed`	false	false	false	null	Completed

💡 A transaction id is earned, never invented. It appears only after the provider reports a final successful status — the Day 11 rule, carried through Day 12's reconciliation and now across a restart. Failed payments carry no id; nothing is fabricated locally.

7. Creation attempts vs status checks — the ledgers still hold

Durability must not weaken any Day 11/12 invariant. Two separate ledgers keep creation attempts and status checks apart, and the restart must add nothing to the first:

// Correct — a status READ of the original attempt, after restart:
CheckPaymentStatus("PAY-001")

// Wrong — a brand-new charge wearing the old key's name:
CreatePayment("PAY-001")

Exactly one creation attempt per key, ever — 6 for 6 keys, all in Instance 1. Zero new creation attempts after restart.
Six status checks after restart — one per reloaded work item, all in Instance 2.
The idempotency key is the payment's identity; it is preserved through the durable store, so the reloaded worker asks about the original attempt, never starts a new one.

💡 Why the key still matters (Days 3–5, again). Without the preserved idempotency key in the durable record, a restarted worker would have nothing to ask about — and any "retry" would become a new charge. Durability preserves the key so follow-up stays a read.

8. The real implementation — proven by running it

Day 13 is implemented as two ISessionScenario classes registered in Program.cs, same as every previous day:

BEFORE: Session13InMemoryWorkLostScenario → command session-13-in-memory-work-lost
AFTER: Session13DurableReconciliationScenario → command session-13-durable-reconciliation

Everything is simulated in-process with standard .NET only — Task.Delay for the hung gateway and the status-read latency, SemaphoreSlim(2) for the bulkhead, a linked CancellationTokenSource for the 300 ms timeout, and System.Text.Json + FileStream for the durable store (write temp → Flush(flushToDisk: true) → File.Move(overwrite: true)). Two separate ApplicationInstance objects model the crash boundary; a single ExternalPaymentProvider survives it. No SQL, no EF, no broker, no scheduler, no packages.

Real BEFORE vs AFTER metrics

Metric (real run)	BEFORE	AFTER
Pending before crash	6 / 6	6 / 6
In-memory work items before crash	6	6
Durably saved (payments / work items)	— / —	6 / 6
In-memory items inherited on restart	0	0
Work items available after restart	0	6 (reloaded from disk)
Status checks after restart	0	6 (one per key)
Resolved as Paid	0	4
Resolved as Failed	0	2
Still pending	6 / 6	0 / 6
Completed work items	0	6 / 6
New payment attempts after restart	0	0
Duplicate charge attempts	0	0
Fake successes	0	0

The AFTER reconciliation pass took ~360 ms — six status checks × ~50 ms plus the durable re-writes. Remembering the promise across a restart costs six small reads and a few file writes, not a single re-charge.

Safety checks from the real AFTER report — all passing

one creation attempt per key, ever → 6 attempts / 6 keys
zero new creations after restart → 0
duplicate charge attempts == 0 → 0
confirmed ⇔ provider reported Paid → fake successes 0
tx ids only from provider, Paid only → all match

💡 Verdicts from the real reports. BEFORE: "THE WORKER KNEW WHAT TO DO — BUT THE CRASH ERASED THE WORK ❌". AFTER: "DURABLE RECONCILIATION — THE PROMISE SURVIVED THE CRASH ✅". Both scenarios exit with code 0 only when every check passes.

Run it yourself

dotnet run --project tools/Wassal.SessionTests -- session-13-in-memory-work-lost
dotnet run --project tools/Wassal.SessionTests -- session-13-durable-reconciliation

Each run prints the per-payment transition table, the summary, the checks, and the verdict, then saves a timestamped report under:

Reports/Session-13/Session-13-In-Memory-Work-Lost-RunReport-<timestamp>.txt
Reports/Session-13/Session-13-Durable-Reconciliation-RunReport-<timestamp>.txt

The AFTER durable file is created in a scenario-specific temp folder and cleaned up at the end of the run — e.g. %LOCALAPPDATA%\Temp\wassal-day13-durable-<random>\reconciliation-store.json. No durable test state is written into the application's own folders.

9. Final takeaway

A correct worker is not enough if its work lives only in memory — a crash erases the obligation.
When a payment becomes PendingConfirmation, persist the payment record and its reconciliation work item together, durably, with a safe atomic write.
A restarted instance must reload its work only from the durable store — no static fields carry state across the boundary.
Our application restarts; the external provider does not. Durability is how our side remembers what the provider still knows.
All Day 11/12 invariants still hold: one creation per key ever, status checks are reads, no duplicate charges, transaction ids only from the provider.
This is outbox-shaped — the direction toward a Transactional Outbox — not a production database-backed outbox.

The three days, side by side:

Day 11:  We do not know yet.
Day 12:  We checked the original attempt and learned the result.
Day 13:  Even if our application crashes, it must remember that the check is still required.

💡 In-memory work disappears. Durable work survives restarts. Pending is a promise — durability makes sure the system remembers the promise.

Six payments went pending and were persisted durably; the process crashed; a fresh instance reloaded six work items from disk and resolved them all — 6 status checks, 4 Paid, 2 Failed, 0 new charges. A later lesson can make this a real transactional outbox with a hosted worker, schedules, and retries; the invariant to carry forward is simpler: the work must survive the process that created it.

Part of the **Fundamentals of Distributed Systems* series — building Wassal, a distributed food-delivery lab, one concept at a time.*

Day 12 — Payment Reconciliation: Resolving the Unknown Safely

mohamed Tayel — Sat, 25 Jul 2026 21:07:35 +0000

Day 11 taught the honest fallback: when the provider times out, say "we do not know yet". But pending-confirmation is a promise that someone will check again. Day 12 keeps that promise — with a reconciliation worker that asks about the original attempt, and never, ever charges again.

🙏 Credit where it's due — This series is my attempt to internalize and share what I learned from Mahmoud Youssef's excellent course, Fundamentals of Distributed Systems on Udemy. The course material, structure, and topic flow are his work; the explanations, code examples, and diagrams in these articles are my own rewrite in my own words. If you find this useful, please consider taking the course — it goes much deeper than these articles can.

🎯 The one idea to take away. Fallback admits uncertainty. Reconciliation resolves uncertainty. Day 11: "We do not know yet." — Day 12: "We checked the original attempt and now know the final result." Resolving means asking about the original attempt — never charging again.

Where we are — after Day 11

Day 11 — Payment Fallback Without Lying: when the payment provider times out, we return an honest degraded response — PendingConfirmation, PaymentConfirmed = false, IsFallback = true, RequiresStatusCheck = true, idempotency key preserved, no fabricated transaction id.
Day 12 (this document): that response contains a to-do item — RequiresStatusCheck = true. Who performs the check? How does a payment move from pending to a real final state? And how do we prove the check never becomes a second charge?

1. Why PendingConfirmation cannot remain forever

PendingConfirmation is honest — and it is also not a final business outcome. Nothing downstream can act on it:

The customer who was charged never gets an order confirmation — money taken, nothing delivered.
The customer who was not charged is never told to retry — the sale is silently lost.
Finance cannot close the books on a payment that is neither paid nor failed.
Support sees "pending" forever and has no answer to give.
✅ Honest fallback (Day 11 · solved) — On timeout, say the truth: pending, not confirmed, check later. Nobody is lied to.
❌ The promise-keeper (Day 12 · still missing) — RequiresStatusCheck = true is a to-do with no owner. Nothing checks, so nothing ever resolves.

⚠️ Honest but incomplete. An unresolved pending payment is a debt of truth. The fallback bought time honestly — reconciliation is how the system pays the debt back.

2. The BEFORE problem — nobody ever checks

The BEFORE scenario (session-12-no-reconciliation) replays Day 11 exactly, then lets time pass:

Phase 1: 6 payments (PAY-001…PAY-006) hit a hung gateway (2000 ms) behind the Day 9 bulkhead with the Day 10 timeout (300 ms). All 6 time out → all 6 become PendingConfirmation, each stored with its idempotency key.
Phase 2: The provider recovers. It now knows the real fate of every original attempt: 4 were actually charged, 2 actually failed. The answers exist and are one status call away.
…forever: Our system has no reconciliation worker. Status checks performed: 0. Resolved: 0. All 6 payments stay pending until the end of time.

From the real run: initial pending 6, provider recovered true, status checks 0, resolved 0, still pending 6, duplicate charges 0.

⚠️ Verdict from the real report: "THE SYSTEM ADMITTED UNCERTAINTY — BUT NEVER RESOLVED IT ❌". Every check in the BEFORE report passes — because the problem it proves is real: honesty without follow-up leaves the business stuck.

3. The AFTER fix — a reconciliation worker

The AFTER scenario (session-12-background-reconciliation) is identical through Phase 1 and the provider recovery. Then one new piece runs — a small, deterministic, in-process reconciliation worker:

Scan the pending store — only payments still in PendingConfirmation are work.
For each, call CheckPaymentStatus(key) — a status read of the original attempt, identified by the same idempotency key. A ~50 ms read, not a charge.
Apply the provider's real answer: Paid → confirmed with the provider's transaction id; Failed → a real failed state, no id.
Clear RequiresStatusCheck — the promise is kept, the payment is final.

From the real run, every payment resolved to the provider's truth:

PAY-001 | PendingConfirmation → Paid
PAY-002 | PendingConfirmation → Paid
PAY-003 | PendingConfirmation → Failed
PAY-004 | PendingConfirmation → Paid
PAY-005 | PendingConfirmation → Failed
PAY-006 | PendingConfirmation → Paid

💡 Kept deliberately small. The worker is a plain awaited loop — no hosted service, no queue, no scheduler, no database. Durability, crash recovery, and retry schedules belong to later lessons. Day 12's invariant is the part worth keeping forever: ask, don't re-charge.

4. Checking a payment is not retrying a payment

This is the sharpest edge of the lesson. Two operations that sound similar and must never be confused:

// Correct — a status READ of the original attempt:
CheckPaymentStatus("PAY-001")

// Wrong — a brand-new charge wearing the old key's name:
CreateNewPayment("PAY-001")

Same idempotency key = same original payment attempt. The key is the payment's identity, carried from Day 11's fallback into Day 12's check.
A status check asks: "what happened to attempt PAY-001?" It moves no money.
A new creation says: "charge the customer." If the original attempt actually succeeded at the provider, this is a double charge.
That is why the implementation keeps two separate ledgers — payment creation attempts and status-check calls — and the report never hides them behind one generic "provider calls" counter.

⚠️ Why the key matters (Days 3–5, again). Without the preserved idempotency key, "check later" is impossible — there is nothing to ask about — and any retry becomes a new charge. The key is what turns follow-up from a gamble into a read.

5. Explicit state transitions

Three states, and every transition is driven by the provider's reported truth — never by our hopes:

PendingConfirmation ──(provider says Paid)───→ Paid
PendingConfirmation ──(provider says Failed)─→ Failed
PendingConfirmation ──(still unresolved)─────→ PendingConfirmation  (check again later)

Final state	PaymentConfirmed	RequiresStatusCheck	IsFallback	TransactionId
`Paid`	true	false	false	real provider id (e.g. TX-PAY-001)
`Failed`	false	false	false	null
`PendingConfirmation` (unresolved)	false	true	true	null

💡 A transaction id is earned, never invented. It appears only after the provider reports a final successful status — exactly the Day 11 rule, carried forward: no fake success, no fabricated id, and a provider-reported failure becomes a real Failed, not a quiet retry.

6. The real implementation — proven by running it

Day 12 is implemented as two ISessionScenario classes registered in Program.cs, same as every previous day:

BEFORE: Session12NoReconciliationScenario → command session-12-no-reconciliation
AFTER: Session12BackgroundReconciliationScenario → command session-12-background-reconciliation

Everything is simulated in-process with standard .NET primitives — Task.Delay for the hung gateway and the status-read latency, SemaphoreSlim(2) for the bulkhead, a linked CancellationTokenSource for the 300 ms timeout, and an in-memory dictionary of provider outcomes (deterministic: 4 Paid with transaction ids, 2 Failed). Two ConcurrentDictionary ledgers count creation attempts and status checks per key, so the safety claims are measured, not asserted.

Real BEFORE vs AFTER metrics

Metric (real run)	BEFORE	AFTER
Initially pending	6 / 6	6 / 6
Provider later recovered	true	true
Payment creation attempts (ever)	6 (6 keys)	6 (6 keys)
Creation attempts during reconciliation	—	0
Status checks performed	0	6 (one per key)
Resolved as Paid	0	4
Resolved as Failed	0	2
Still pending	6 / 6	0 / 6
Duplicate charge attempts	0	0
Fake successes	0	0
Total scenario duration	938 ms	1250 ms

The AFTER run is ~300 ms longer — that is the reconciliation pass itself: 6 status checks × ~50 ms. Resolving the unknown costs six small reads, not one risky re-charge.

Safety checks from the real AFTER report — all passing

one creation attempt per key, ever → 6 attempts / 6 keys
zero creations during reconciliation → 0
duplicate charge attempts == 0 → 0
confirmed ⇔ provider reported Paid → fake successes 0
transaction ids only from provider → all match
failed payments carry no tx id → none do

💡 Verdicts from the real reports. BEFORE: "THE SYSTEM ADMITTED UNCERTAINTY — BUT NEVER RESOLVED IT ❌". AFTER: "BACKGROUND RECONCILIATION — THE UNKNOWN WAS RESOLVED, AND NOBODY WAS CHARGED TWICE ✅". Both scenarios exit with code 0 only when every check passes.

Run it yourself

dotnet run --project tools/Wassal.SessionTests -- session-12-no-reconciliation
dotnet run --project tools/Wassal.SessionTests -- session-12-background-reconciliation

Each run prints the per-payment transition table, the summary, the checks, and the verdict, then saves a timestamped report under:

Reports/Session-12/Session-12-No-Reconciliation-RunReport-<timestamp>.txt
Reports/Session-12/Session-12-Background-Reconciliation-RunReport-<timestamp>.txt

7. Final takeaway

PendingConfirmation is honest, but it is not a final business outcome — it is a promise to check again.
Reconciliation keeps the promise by asking about the original attempt — CheckPaymentStatus(key), never CreateNewPayment(key).
The same idempotency key means the same payment attempt. One creation per key, ever; status checks are reads.
A payment becomes Paid only when the provider reports it; a provider-reported failure becomes a real Failed; an unresolved answer may stay pending — to be checked again.
Transaction ids come from the provider or not at all.

The two days, side by side:

Day 11:  We do not know yet.
Day 12:  We checked the original attempt and now know the final result.

💡 Fallback admits uncertainty. Reconciliation resolves uncertainty. Together they let a payment system survive a provider outage without ever lying — and without ever charging twice.

Pending is a promise to check again. Day 12 keeps the promise. Six payments went pending honestly, the provider recovered, and one small worker resolved them all — 6 status checks, 4 Paid, 2 Failed, 0 new charges. A later lesson can make the worker durable (schedules, queues, crash recovery); the invariant to carry forward is simpler: ask, don't re-charge.

Part of the **Fundamentals of Distributed Systems* series — building Wassal, a distributed food-delivery lab, one concept at a time.*

Day 11 — Payment Fallback: Without Lying

mohamed Tayel — Sat, 25 Jul 2026 20:27:57 +0000

Bulkhead limits how many. Circuit breaker stops the storm. Timeout stops the wait. But when the payment provider still cannot give a final answer, one business question remains: what do we tell the caller? For payments, the fallback must be honest — never a fake success.

🙏 Credit where it's due — This series is my attempt to internalize and share what I learned from Mahmoud Youssef's excellent course, Fundamentals of Distributed Systems on Udemy. The course material, structure, and topic flow are his work; the explanations, code examples, and diagrams in these articles are my own rewrite in my own words. If you find this useful, please consider taking the course — it goes much deeper than these articles can.

🎯 The one idea to take away. The safest fallback is not a fake answer. It is an honest state. For payments, unknown is a valid state. Do not convert unknown into success — and do not convert it into failure either. Say what is true: we do not know yet.

Where we are — after Day 10

Day 8 — Circuit Breaker: when a dependency keeps failing, stop hammering it. Fail fast and give it room to recover.
Day 9 — Bulkhead Isolation: give each dependency its own pool of slots, so one slow dependency cannot starve the others.
Day 10 — Timeout & Cancellation: bound how long a call may hold a slot. Stop waiting, cancel, release.
Day 11 (this document): all of that protects the system. Now we answer for the caller: what should the system return when the dependency still cannot give a final answer?

1. Three protections in place — one question still open

After Days 8–10, our calls to a payment provider are well protected. Each mechanism answers its own question:

✅ Bulkhead (Day 9 · how many?) — At most N payment calls run at once. One slow dependency cannot starve the rest.
✅ Circuit breaker (Day 8 · keep trying?) — When the provider keeps failing, stop hammering it and fail fast for a while.
✅ Timeout (Day 10 · how long?) — A call may wait ~300 ms, then it is cancelled and its slot is released.
❌ Fallback (Day 11 · what to return?) — The timeout fired. The breaker is open. What do we tell the caller? Nothing so far decides that.

⚠️ The gap. Timeout decides when to stop waiting. It does not decide what to say after you stopped. All three mechanisms are infrastructure answers — the fallback is a business answer, and it must match the business risk.

2. Why a "normal" fallback is not safe for payments

For a read-only, low-risk feature, a fallback can be simple and nobody gets hurt. Product recommendations are the classic example:

return cached recommendations,
return the popular items list,
or even return an empty list.

Worst case, the customer sees slightly generic suggestions. The business risk is near zero — so a cheap default is a perfectly good fallback.

Payments are the opposite end of the risk scale. A payment call moves money and changes state. And a payment timeout is ambiguous by nature: the provider may have already charged the card — we simply never received the confirmation. There is no "safe default value" for money.

Question	Recommendations	Payments
What does the call do?	Reads data	Moves money, changes state
Is a default value harmless?	Yes — popular items are fine	No — there is no default for money
What does a timeout mean?	No suggestions this time	Unknown — the charge may have happened
Cost of a wrong fallback	A slightly generic page	Double charge, broken reconciliation, lost trust
Good fallback	Cached / popular / empty	An honest pending state

⚠️ Fallback must match the business risk. The same pattern — "return something instead of an error" — is harmless for recommendations and dangerous for payments. Copying a read-side fallback onto a payment path is how systems start lying.

3. The wrong solution — fake success

The tempting shortcut: the provider timed out, the customer is waiting, so just… pretend it worked and sort it out later.

PaymentStatus = Paid
TransactionId = FAKE-123
Message       = Payment completed successfully

This response is a lie, and every part of the business eventually pays for it:

The customer may receive an order without a real payment — the charge never happened, but the order shipped.
Finance reconciliation breaks — FAKE-123 matches nothing at the provider. Someone has to untangle it by hand.
Support cannot know the real payment state — the system itself no longer knows which payments are real.
Retrying later may double-charge the customer — the original attempt may have succeeded at the provider; a "retry" creates a second real charge.
The system loses truth — once fake data enters your records, every report, refund, and audit built on top of it inherits the lie.

⚠️ Do not convert unknown into success. A fake Paid does not remove the uncertainty — it hides it inside your own database, which is the worst place for it to live.

4. The correct solution — explicit pending confirmation

The honest answer has three properties: it does not say the payment failed, it does not say the payment succeeded, and it clearly marks itself as a degraded response that needs a follow-up.

PaymentStatus       = PendingConfirmation
PaymentConfirmed    = false
IsFallback          = true
RequiresStatusCheck = true
Message             = Payment provider did not confirm in time.
                      Payment status must be checked later.

Read it back in plain words: "We asked the provider to charge the card. We did not hear back in time. We do not know yet — and here is what to do about it." That is the honest distributed systems answer.

As a response model, it stays small and boring on purpose:

{
  "orderId": 1001,
  "paymentStatus": "PendingConfirmation",
  "paymentConfirmed": false,
  "isFallback": true,
  "requiresStatusCheck": true,
  "message": "Payment provider did not confirm in time. Please check payment status later."
}

Field	Value	Why it is there
`paymentStatus`	PendingConfirmation	Not success, not failure — the true state.
`paymentConfirmed`	false	Nothing downstream (order confirmation, shipping) may proceed as if money moved.
`isFallback`	true	The response is clearly marked as degraded — the UI and the logs can both tell.
`requiresStatusCheck`	true	An actionable next step, not a dead end.
`message`	"…check payment status later."	Something honest the caller can show a human.

💡 For payments, unknown is a valid state. PendingConfirmation gives uncertainty a name and a home in your model — instead of forcing it, wrongly, into Paid or Failed.

5. What the caller should do next

A pending response is only useful if the caller knows what to do with it. Three simple rules:

Show the truth to the user. "Your payment is being confirmed — we will update you shortly." Not "Payment successful", not "Payment failed".
Do not trigger success-only side effects. No order confirmation email, no shipping label, no loyalty points — those wait for paymentConfirmed = true.
Check status later, or retry with the same idempotency key. The follow-up asks about the same payment attempt — it never starts a new one.

💡 Kept deliberately simple. In a production system the "check later" part often becomes a background reconciliation job that polls the provider and settles pending payments automatically. That deserves its own lesson — Day 11 stays focused on the honest response itself.

6. Why idempotency still matters

The pending state only works because of an old friend from Days 3–5: the Idempotency-Key. The fallback must preserve it, because the key is what makes "try again later" safe:

Same order + same idempotency key = same payment attempt. Always.
A retry checks or continues that same attempt — it never creates a second one.
Never create multiple real payment attempts for one customer action.
The pending status must be replayable: asking again returns the same pending result, or the latest known status — not a new charge.

The idempotent retry story, in two beats:

First request: OrderId = 1001, Idempotency-Key = abc-123 → payment provider timeout → return PaymentPendingConfirmation.
Second request: Same Idempotency-Key = abc-123 → do not create a second payment → return the same pending result, or the latest known status.

⚠️ The double-charge trap. A retry with a new key is a brand-new payment in the provider's eyes. If the first (timed-out) attempt actually succeeded, the customer is now charged twice. The idempotency key is the difference between "asking again" and "charging again".

7. How this connects to the previous days

Each day answered one question. Day 11 adds the final, business-facing one:

Day	Mechanism	The question it answers
Days 3–5	Idempotency	How do we make retries safe?
Day 8	Circuit breaker	Should we keep trying a failing dependency?
Day 9	Bulkhead	How many calls may run at once?
Day 10	Timeout + cancellation	How long may a call wait?
Day 11	Honest fallback	What do we return when there is still no final answer?

Notice the shape: Days 8–10 protect the system and never touch the response. Day 11 protects the truth of the response — and it leans on Days 3–5 to make the follow-up safe. All of it composes: the bulkhead and timeout free the slot quickly, the breaker stops the storm, the idempotency key keeps the retry safe, and the honest fallback tells the caller exactly where things stand.

8. The real implementation — proven by running it

This lesson is not just a diagram — it is implemented and verified in the test runner as two ISessionScenario classes, registered in Program.cs like every previous day:

BEFORE: Session11NoSafeFallbackScenario → command session-11-no-safe-fallback
AFTER: Session11SafeFallbackScenario → command session-11-safe-fallback

How the failure is simulated

Everything runs in-process — no server, no database, no packages. The payment gateway is a bounded Task.Delay(2000 ms) that never answers in time. Each call runs through the Day 9 bulkhead (SemaphoreSlim(2)) with the Day 10 per-call timeout: a linked CancellationTokenSource with CancelAfter(300 ms). Six payment operations are fired, each carrying its own idempotency key: PAY-001 … PAY-006. In both scenarios all 6 gateway calls time out — the difference is only what the caller is told.

What BEFORE demonstrates

The timeout path returns a bare hard failure: Result = failed, Reason = payment-timeout — with no pending state, no RequiresStatusCheck, no retry advice, and the idempotency key not echoed back. Every infrastructure check passes (slots freed at ~305 ms, whole run ~920 ms), yet user-visible graceful responses are 0 / 6. The system is protected; the caller hits an ambiguous dead end.

What AFTER changes

One thing only: the catch (OperationCanceledException) block builds the honest fallback instead of a bare failure —

response = new PaymentResponse
{
    Result              = "pending-confirmation",
    PaymentConfirmed    = false,
    IsFallback          = true,
    Source              = "safe-payment-fallback",
    Reason              = "primary-timeout",
    RequiresStatusCheck = true,
    RetryAdvice         = "retry-with-same-idempotency-key",
    IdempotencyKey      = idempotencyKey,   // preserved and echoed back
    TransactionId       = null,             // never fabricated
};

A gateway-attempt ledger (a ConcurrentDictionary keyed by idempotency key) counts every real gateway call, so the report can prove there was exactly one attempt per key — no silent automatic retry.

What the real run proves

From the generated reports (both scenarios exit with code 0):

Metric (real run)	BEFORE	AFTER
Payment gateway timeouts	6 / 6	6 / 6 (still down)
Hard failures returned to caller	6 / 6	0 / 6
Pending-confirmation responses	0 / 6	6 / 6
Fallback clearly marked (`IsFallback`)	0 / 6	6 / 6
Fake successes / confirmed payments / fabricated tx ids	0 / 0 / 0	0 / 0 / 0
Idempotency keys preserved & echoed	0 / 6	6 / 6
RequiresStatusCheck responses	0 / 6	6 / 6
Duplicate charge attempts	0	0 (6 attempts / 6 keys)
Avg slot-hold time	305 ms	300 ms
Total scenario duration	920 ms	905 ms
User-visible graceful responses	0 / 6 (0%)	6 / 6 (100%, all degraded)

Hold time is ~300–305 ms, but average response time is higher (~600 ms) because later waves wait in the bulkhead queue first: response time = wait + hold.

The AFTER report ends with dedicated safety checks, all passing on the real run: fake successes == 0, confirmed payments == 0 (timed out), fabricated transaction ids == 0, every fallback marked IsFallback=true → 6 / 6, idempotency key preserved & echoed → 6 / 6, and exactly one gateway attempt per key → 6 attempts / 6 keys, duplicates 0.

💡 Verdicts from the real reports. BEFORE: "TIMEOUT PROTECTED THE SYSTEM — BUT EVERY PAYMENT ENDED IN AN AMBIGUOUS HARD FAILURE ❌". AFTER: "SAFE PAYMENT FALLBACK — GATEWAY STILL DOWN, NOBODY WAS LIED TO ✅". Same hung gateway, same timeout — only the honesty of the answer changed.

Run it yourself

dotnet run --project tools/Wassal.SessionTests -- session-11-no-safe-fallback
dotnet run --project tools/Wassal.SessionTests -- session-11-safe-fallback

Each run prints the per-operation table, summary, checks, and verdict, then saves a timestamped report under:

Reports/Session-11/Session-11-No-Safe-Fallback-RunReport-<timestamp>.txt
Reports/Session-11/Session-11-Safe-Fallback-RunReport-<timestamp>.txt

9. Final takeaway

A fallback is not always "return some default data". Fallback must match the business risk.
For low-risk reads (recommendations), a cached or popular-items default is fine.
For payments, a timeout means unknown — and unknown is a valid state. Give it a name: PendingConfirmation.
Never return Paid, a fake transaction id, or "payment completed" without real confirmation from the provider.
Preserve the idempotency key so that checking or retrying later continues the same attempt instead of charging twice.

💡 Do not convert unknown into success. The system that admits "we do not know yet" keeps its truth — and a system that keeps its truth can always recover. A system that lies to its own database cannot.

The safest fallback is not a fake answer. It is an honest state. Bulkhead, breaker, and timeout protect the system. The honest fallback protects the truth. For payments that means: not success, not failure — pending, clearly marked, safely retryable. A later lesson can add background reconciliation and status checking on top of this foundation.

Part of the **Fundamentals of Distributed Systems* series — building Wassal, a distributed food-delivery lab, one concept at a time.*

Day 10 — Timeout & Cancellation: Do Not Wait Forever

mohamed Tayel — Sat, 25 Jul 2026 20:27:19 +0000

Day 9's bulkhead limits how many calls run at once — but never how long a call may hold a slot. When a dependency hangs, its slots stay occupied far too long and everyone queues behind them. First we prove the gap, then we close it with a per-call timeout and cooperative cancellation.

🙏 Credit where it's due — This series is my attempt to internalize and share what I learned from Mahmoud Youssef's excellent course, Fundamentals of Distributed Systems on Udemy. The course material, structure, and topic flow are his work; the explanations, code examples, and diagrams in these articles are my own rewrite in my own words. If you find this useful, please consider taking the course — it goes much deeper than these articles can.

🎯 The one idea to take away: Bulkhead answers "how many calls can run at once?" — Timeout answers "how long is a call allowed to hold a slot?" A bulkhead with no timeout still lets one hung call sit in a slot for as long as the dependency hangs. Limiting concurrency is only half the job.

Part 1 — The problem: a hung dependency holds its slots forever

Two different questions

Protecting a dependency call has two independent dimensions. Day 9 answered the first. Day 10 is about the second — the one a bulkhead leaves wide open.

Dimension	Status	What it caps
✅ Bulkhead — how many?	Day 9 · solved	"At most 2 Payment calls run at once." Concurrency is capped by `SemaphoreSlim(2)`.
❌ Timeout — how long?	Day 10 · still missing	"A call may hold a slot for at most 300 ms." Nothing here yet — a hung call holds its slot for 2000 ms.

💡 The gap. A bulkhead caps concurrency, but a hung call still occupies its slot for the dependency's full hang time. Two slots held for 2000 ms each means the next calls simply wait — the bulkhead can't help, because the slow calls are inside the slots, not outside them.

Scenario setup — one hung dependency, its own bulkhead

Everything is simulated in-process (no real server, no database, no external packages). We deliberately keep Day 10 focused on one dependency — Payment — and its own bulkhead. No Inventory, no Notification this time; the whole lesson is about slot-hold time.

Dependency 💳 Payment is hung — every call takes 2000 ms (simulated with a bounded await Task.Delay(2000, token)).
It runs through its own bulkhead: paymentPool = new SemaphoreSlim(2) — carried straight over from Day 9.
We fire 6 Payment operations. There is no per-call timeout.

The Payment bulkhead has 2 slots, and both get stuck on a 2000 ms hang. Payment #1 and #2 hold the slots; Payments #3, #4, #5, and #6 wait in line for a slot to free — which won't happen for ~2000 ms.

💡 Why this is deterministic. 6 calls through 2 slots = 3 waves. Each wave holds both slots for the full 2000 ms hang, so wave 2 starts at ~2000 ms and wave 3 at ~4000 ms. Every number below falls out of that simple arithmetic.

Timeline — three slow waves

~0 ms — Wave 1 — Payment #1 and #2 grab both slots and start their 2000 ms hang. #3–#6 queue.
~2000 ms — Wave 2 — #1 and #2 finally finish and release. #3 and #4 take the slots and hang for another 2000 ms.
~4000 ms — Wave 3 — #5 and #6 finally get in and hang for 2000 ms more.
~6000 ms — Everything is done. Total run ≈ 6034 ms for work that should have taken a fraction of that.

💡 The slots never free early. Because there's no timeout, a slot is only released when the 2000 ms hang finally returns. Nothing can shorten it. 0 of 6 slots were released early.

The star metric — slot-hold time

The number that matters in Day 10 is slot-hold time: how long a call keeps its bulkhead slot from the moment it acquires it to the moment it releases it. With no timeout, that equals the full hang — about 2007 ms on average. The calls behind it wait ~2013 ms on average, and up to 4020 ms in the last wave.

⚠️ The slot is the bottleneck. A hung call holds its slot ~2007 ms, so the calls behind it wait ~2013 ms on average and up to 4020 ms. The bulkhead limited concurrency to 2 — it did nothing about the 2000 ms hold.

Per-operation results (from the real run):

#	Dependency	Timeout	Work	Wait	Hold	Total	Reason
1	Payment	none	2000	0	2021	2021	completed
2	Payment	none	2000	0	2018	2018	completed
3	Payment	none	2000	2020	2001	4021	completed
4	Payment	none	2000	2020	2000	4020	completed
5	Payment	none	2000	4020	2000	6021	completed
6	Payment	none	2000	4020	2000	6021	completed

Every call "completed" — but only after holding its slot for ~2000 ms. Waits climb 0 → 2020 → 4020 ms across the three waves.

📊 Report numbers (from the real run):

Metric	Value
Total operations	6
Payment bulkhead size	2 slots
Hung dependency duration (each)	2000 ms
Timeout	none
Average wait (queue for a slot)	2013 ms
Max wait	4020 ms
★ Average slot-hold time	2007 ms
Max slot-hold time	2021 ms
Timed-out operations	0 / 6
Slots released early	0 / 6
Total scenario duration	6034 ms

Diagnostic verdict: NO TIMEOUT — HUNG DEPENDENCY HELD ITS SLOTS AND STALLED EVERYTHING ✅

What this proves:

The bulkhead did its Day 9 job — never more than 2 Payment calls ran at once.
But with no timeout, each hung call held its slot for the full 2000 ms — slots never freed early.
So the 6 calls serialized into 3 slow waves, later calls waited up to 4020 ms, and the whole run took 6034 ms.
Limiting how many calls run is not enough. You also have to limit how long each one may hold a slot.

Part 2 — The fix: timeout + cooperative cancellation

Both dimensions, together

Day 9 and Day 10 are two halves of the same protection. Once you have both, a dependency can neither run away with all your concurrency nor sit forever in the slots it holds.

Dimension	When	What it caps
✅ Bulkhead — how many?	Day 9	At most 2 Payment calls run at once — `SemaphoreSlim(2)`.
✅ Timeout — how long?	Day 10 · this step	A call may hold a slot for at most ~300 ms, then it is cancelled and the slot is released.

✅ Two knobs, two questions. Concurrency is capped by the bulkhead; hold time is capped by the timeout. Neither replaces the other — you want both.

The fix — timeout + cooperative cancellation

The dependency is unchanged: paymentPool = new SemaphoreSlim(2), still 6 calls, each still hanging for 2000 ms. The only new thing is a 300 ms per-call timeout built from a linked CancellationTokenSource.

Link a token from the outer (Ctrl+C) token so external cancellation still works.
CancelAfter(300 ms) — the linked source will fire the token when the call overruns.
Pass the linked token into the dependency call (Task.Delay(HungWorkMs, timeoutCts.Token)), so cancellation is cooperative — the call actually stops when asked.
Catch the cancellation and tell our timeout apart from an external cancel, so we report timed-out vs aborted correctly.
Release the slot in finally — on every path, success or timeout. This is what frees the bulkhead early.

using var timeoutCts = CancellationTokenSource.CreateLinkedTokenSource(ct);
timeoutCts.CancelAfter(TimeoutMs);

try
{
    await Task.Delay(HungWorkMs, timeoutCts.Token);
}
catch (OperationCanceledException) when (timeoutCts.IsCancellationRequested && !ct.IsCancellationRequested)
{
    // our timeout fired
    result = "timed-out";
}
finally
{
    paymentPool.Release();
}

⚠️ Cancellation must be cooperative. Passing the token into Task.Delay is what makes the hung call actually stop at 300 ms. A timeout you don't pass down is just a stopwatch that nobody obeys — the slot would still be held for 2000 ms.

💡 Always release in finally. If the slot were only released on the success path, a timed-out call would keep its slot for the full hang — silently recreating the Part 1 bug.

Timeline — waves that move fast

~0 ms — Wave 1 — #1 and #2 take both slots. At ~300 ms their timeout fires, they abandon and release.
~300 ms — Wave 2 — #3 and #4 immediately take the freed slots, time out at ~300 ms more, release.
~600 ms — Wave 3 — #5 and #6 get in, time out at ~300 ms, release.
~920 ms — All 6 done — each returned a deterministic timed-out result. Total run ≈ 920 ms vs 6034 ms before.

✅ The dependency is still hung. Every call would still take 2000 ms if we let it. We simply stop waiting at 300 ms — so the slots free ~6.5× faster and the whole run finishes in under a second.

The star metric — slot-hold time: 2007 ms → 306 ms

Same metric as Part 1, now bounded by the timeout. A slot is held only until the 300 ms timeout fires, not until the 2000 ms hang returns. Slot-hold drops from 2007 ms to 306 ms, the average wait falls to 309 ms, and the whole scenario finishes in 920 ms.

✅ ~6.5× less slot-hold. The hang is unchanged at 2000 ms, but the slot is held for only ~306 ms — so waits collapse to ~309 ms and the whole run drops from 6034 ms to 920 ms.

Per-operation results (from the real run):

#	Dependency	Timeout	Work	Wait	Hold	Total	Reason
1	Payment	300	2000	0	311	311	timed-out
2	Payment	300	2000	0	310	310	timed-out
3	Payment	300	2000	310	308	618	timed-out
4	Payment	300	2000	310	307	618	timed-out
5	Payment	300	2000	618	299	918	timed-out
6	Payment	300	2000	618	299	918	timed-out

Every call returned a deterministic timed-out result after ~300 ms — not a hang. Waits climb only 0 → 310 → 618 ms, because slots free quickly.

📊 Report numbers (from the real run):

Metric	Value
Total operations	6
Payment bulkhead size	2 slots
Hung dependency duration (each)	2000 ms (still hung)
Timeout	300 ms
Average wait (queue for a slot)	309 ms
Max wait	618 ms
★ Average slot-hold time	306 ms
Max slot-hold time	311 ms
Timed-out operations	6 / 6
Slots released	6 / 6
Total scenario duration	920 ms

⚖️ Before vs after — same hang, bounded hold:

Metric	BEFORE	AFTER
★ Avg slot-hold time	2007 ms	306 ms
Avg wait (queue for a slot)	2013 ms	309 ms
Max wait	4020 ms	618 ms
Timed-out operations	0 / 6	6 / 6
Scenario duration	6034 ms	920 ms
Hung dependency (each)	2000 ms	2000 ms

The hang is identical in both runs — 2000 ms. Timeout did not fix the dependency; it bounded how long a call may hold a slot, so slot-hold dropped from 2007 ms to 306 ms and the run went from 6034 ms to 920 ms.

Proof verdict: TIMEOUT & CANCELLATION — SLOTS RELEASED, SYSTEM STAYED RESPONSIVE ✅

What this proves:

The dependency is still hung — every call would still take 2000 ms if allowed. Timeout did not heal it.
But each call now holds its slot for only ~306 ms before the timeout cancels it and finally releases the slot.
All 6 / 6 calls timed out deterministically and every slot was released — no call hung, no slot leaked.
The run dropped from 6034 ms → 920 ms. Callers fail fast instead of waiting forever, and the bulkhead recovers instead of strangling itself.

💡 The lesson behind the lesson: a timeout is a promise you make to yourself about how long you'll wait — not a promise the dependency will be quick. Pair it with the Day 9 bulkhead and you've capped both how many calls run and how long each may hold a slot.

Run it yourself

Run the BEFORE / diagnostic scenario:

dotnet run --project tools/Wassal.SessionTests -- session-10-no-timeout

Then run the AFTER / proof scenario:

dotnet run --project tools/Wassal.SessionTests -- session-10-timeout

Each prints the per-operation table, the summary, the checks, and the verdict — then exits with code 0 when the scenario is proven. Timestamped reports are saved under:

Reports/Session-10/Session-10-No-Timeout-RunReport-<timestamp>.txt
Reports/Session-10/Session-10-Timeout-RunReport-<timestamp>.txt

💡 No setup needed. The hung dependency, its bulkhead, and the timeout all live inside the test runner. The 2000 ms hang is a bounded delay (never a real infinite wait), and a scenario-level safety cap guards against anything hanging the runner.

The dependency stayed hung at 2000 ms, but our caller stopped waiting at 300 ms, cancelled cooperatively, and released the slot. Bulkhead limits how many; timeout limits how long. Do not wait forever.

Part of the **Fundamentals of Distributed Systems* series — building Wassal, a distributed food-delivery lab, one concept at a time.*

Day 9 — Bulkhead Isolation: Contain the Slow Dependency

mohamed Tayel — Sat, 25 Jul 2026 20:27:15 +0000

When every kind of work shares one small pool of slots, a single slow dependency grabs them all — and perfectly healthy work waits behind it. First we build the ship without bulkheads to watch the water spread, then we add watertight walls that contain the leak.

🙏 Credit where it's due — This series is my attempt to internalize and share what I learned from Mahmoud Youssef's excellent course, Fundamentals of Distributed Systems on Udemy. The course material, structure, and topic flow are his work; the explanations, code examples, and diagrams in these articles are my own rewrite in my own words. If you find this useful, please consider taking the course — it goes much deeper than these articles can.

🎯 The one idea to take away: A slow dependency should never be allowed to consume all of your shared resources. If all your work shares one pool of slots, then one slow thing fills the pool — and everything else, healthy or not, has to wait. Bulkhead isolation gives each dependency its own pool, so a slow one can only flood its own compartment.

Part 1 — The problem: one shared pool for everything

Real-world analogy — the ship's compartments

A ship's hull is divided into sealed compartments called bulkheads. If one compartment springs a leak, the watertight walls keep the water trapped there — the other compartments stay dry, and the ship stays afloat.

A ship with no bulkheads is the opposite: one leak, and water spreads through the entire hull until the whole ship sinks — even though only one small area was actually damaged.

In software it's the same idea. Without resource isolation, one slow dependency "floods" the shared pool and the delay spreads to unrelated, healthy work. Today we build the ship without bulkheads on purpose — to see the water spread.

One shared pool for everything

Client-side, work usually runs against a limited pool of worker slots (threads, connections, permits). The mistake is letting every kind of dependency call compete for the same pool.

Different dependency calls share the same limited slots.
If one dependency turns slow, its calls sit in those slots for a long time.
Healthy calls — which don't even use the slow dependency — can't get a slot, so they wait.
The failure spreads from one slow dependency to unrelated healthy work.

💡 The trap: the healthy work isn't slow because it is broken. It's slow because it had to queue behind something else that was broken — all because they shared one pool.

Scenario setup — three kinds of work, one pool

We simulate three kinds of client work, all in-process (no real server, no database). Two are perfectly healthy; one is slow.

Dependency	Role	Operations	Work each
💳 Payment	SLOW dependency	6	700 ms
📦 Inventory	healthy & fast	3	60 ms
🔔 Notification	healthy & fast	3	60 ms

The anti-pattern: one global pool of 4 slots. All three kinds of work share a single SemaphoreSlim(4). Payment starts first and grabs all four slots — so everyone else queues, healthy or not. Payments #5 and #6 queue too, along with all 3 Inventory and all 3 Notification operations, none of which can get a slot for ~700 ms.

💡 Why this is deterministic. Payment is launched first and holds all 4 slots for 700 ms. Healthy work only starts contending ~50 ms later — so it cannot possibly get a slot until a Payment call finishes. Every healthy op is guaranteed to wait.

Timeline — watch the water spread

Here is what actually happens, in order:

~0 ms — 6 Payment ops launch. The first 4 grab all shared slots instantly; Payments #5 and #6 queue.
~50 ms — 6 healthy ops (Inventory + Notification) arrive — but every slot is already full of Payment. They join the queue.
~50–700 ms — The pool stays full of 700 ms Payment work. Healthy work — which only needs 60 ms — sits idle, waiting.
~700 ms — The first Payments finish and free slots. Healthy work finally starts… having already waited ~650+ ms for nothing.
~725–860 ms — Healthy ops run their tiny 60 ms of work and finish — but their total time was dominated by waiting, not working.

💡 The leak spread. Inventory and Notification were never broken. They were healthy the whole time — they just drowned in the same pool as slow Payment.

The cost — 60 ms of work, ~726 ms of waiting

A healthy operation should take about 60 ms — that's all the work it does. Instead, it spent most of its life stuck in the queue: 726 ms on average, up to 792 ms in the worst case.

⚠️ ~12× slower than it should be. Healthy work that needed 60 ms waited around 726 ms — almost all of that time was pure queuing behind slow Payment calls, not real work.

📊 Report numbers (from the real run):

Metric	Value
Total operations	12
Shared pool size	4 slots
💳 Payment operations	6
📦🔔 Healthy operations (Inventory + Notification)	6
Payment work duration (each)	700 ms
Inventory work duration (each)	60 ms
Notification work duration (each)	60 ms
Average wait — healthy work	726 ms
Max wait — healthy work	792 ms
Healthy ops delayed significantly	6 / 6

All 6 healthy operations completed successfully — they were never broken. They were simply made ~12× slower by sharing a pool with slow Payment work.

Diagnostic verdict: NO BULKHEAD — SLOW DEPENDENCY DELAYED HEALTHY WORK ✅

What this proves:

The healthy work was not slow because Inventory or Notification were broken — they did their 60 ms of work just fine.
It was slow because all dependency calls shared the same pool with the slow Payment calls.
One slow dependency was enough to delay unrelated healthy work by an order of magnitude.
This is a contagion problem: the failure (slowness) leaked from one place to everywhere through the shared resource.

Part 2 — The fix: give each dependency its own bulkhead

What bulkhead isolation means

In software, a bulkhead is the ship-compartment idea applied to resources:

Each dependency gets its own isolated pool of slots, instead of sharing one global pool.
A slow dependency can only ever consume its own slots — it can't touch anyone else's.
The goal is not to make Payment faster. Payment is still slow.
The goal is to stop Payment from slowing down Inventory and Notification.

💡 Watertight walls between dependencies. If a slow dependency can only flood its own compartment, it can't take the rest of the system down with it. Bulkhead isolation is about limiting the blast radius.

The AFTER setup — three isolated bulkheads

The work is identical to Part 1 — same dependencies, same durations, same counts. The only thing that changes is the pools: instead of one shared pool, each dependency gets its own.

var paymentPool      = new SemaphoreSlim(2);   // Payment's own bulkhead
var inventoryPool    = new SemaphoreSlim(2);   // Inventory's own bulkhead
var notificationPool = new SemaphoreSlim(2);   // Notification's own bulkhead

Payment operations only ever take a slot from paymentPool. Inventory only uses inventoryPool. Notification only uses notificationPool. Nothing crosses between them.

💡 Same head start, opposite result. Just like Part 1, Payment launches first and gets a ~50 ms head start. But this time it fills only its own 2-slot bulkhead — so when the healthy work arrives, the inventory and notification bulkheads are still wide open. The head start no longer matters.

Where the slow work goes now

Before — one shared pool: Payment fills the shared 4-slot pool; everyone else — healthy or not — queues behind it.

After — three isolated bulkheads: each dependency has its own 2-slot compartment. Payment fills its own bulkhead and Payments #3–#6 queue inside the Payment bulkhead only. Meanwhile the Inventory and Notification bulkheads have free slots, so healthy work runs immediately.

💡 The leak is contained. The slow Payment "flood" can only fill the Payment compartment. Inventory and Notification stay dry — their slots were never Payment's to take.

Timeline — the healthy work no longer waits

~0 ms — 6 Payment ops launch. The first 2 fill the Payment bulkhead; Payments #3–#6 queue inside the Payment bulkhead.
~50 ms — 6 healthy ops arrive. Their bulkheads are empty, so the first 2 Inventory + first 2 Notification ops start immediately.
~60–130 ms — Healthy ops finish their 60 ms of work. The 3rd Inventory + 3rd Notification op wait only ~60 ms — for their own bulkhead, not for Payment.
~130 ms — All 6 healthy ops are done — while Payment is still grinding through its own queue.
~2120 ms — Payment finally finishes its 6 × 700 ms of work (queued behind itself, 2 at a time). Still slow — but it hurt no one else.

💡 The flood stayed in one compartment. Payment took ~2.1 seconds, exactly as slow as before. But the healthy work finished in ~130 ms — it never even noticed Payment was struggling.

The win — 726 ms of waiting → 21 ms

A healthy operation does about 60 ms of work. In Part 1 it spent most of its life queued behind Payment. With bulkheads, the wait nearly vanishes: the average healthy wait drops to 21 ms, and even the worst case (62 ms) is below the operation's own work duration.

✅ ~35× less waiting. Healthy work went from ~726 ms of average wait down to just 21 ms. None of the 6 healthy ops was delayed significantly (threshold: wait ≥ 200 ms).

📊 Report numbers (from the real run):

Metric	Value
Total operations	12
💳 Payment bulkhead size	2 slots
📦 Inventory bulkhead size	2 slots
🔔 Notification bulkhead size	2 slots
💳 Payment operations	6
📦🔔 Healthy operations (Inventory + Notification)	6
Payment work duration (each)	700 ms (still slow)
Healthy work duration (each)	60 ms
Average wait — Payment (its own bulkhead)	705 ms
Average wait — healthy work	21 ms
Max wait — healthy work	62 ms
Healthy ops delayed significantly	0 / 6
Total scenario duration	2120 ms

⚖️ Before vs after — same work, isolated pools:

Metric (healthy work)	BEFORE	AFTER
Avg wait — healthy	~726 ms	21 ms
Max wait — healthy	~792 ms	62 ms
Healthy ops delayed significantly	6 / 6	0 / 6
Payment still slow?	700 ms	700 ms

Payment is exactly as slow in both runs — that's the point. Bulkhead isolation didn't fix Payment; it stopped Payment from hurting the healthy work. The healthy wait collapsed from ~726 ms to 21 ms.

Proof verdict: BULKHEAD ISOLATION — SLOW PAYMENT WAS CONTAINED ✅

What this proves:

The slow Payment work still ran slowly (700 ms each, ~2.1 s total) — the bulkhead does not fix the slow dependency.
But Payment was contained to its own pool — it queued only behind itself, never in front of healthy work.
The healthy work waited an average of just 21 ms (max 62 ms), down from ~726 ms — 0 of 6 healthy ops were delayed significantly.
One slow dependency could no longer leak across a shared resource and drag down unrelated work.

💡 The lesson behind the lesson: isolation is about limiting blast radius. You may not be able to make a dependency fast — but you can make sure its slowness stays its own problem.

Run it yourself

Run the BEFORE / diagnostic scenario:

dotnet run --project tools/Wassal.SessionTests -- session-09-no-bulkhead

Then run the AFTER / proof scenario:

dotnet run --project tools/Wassal.SessionTests -- session-09-bulkhead

Each prints the per-operation table, the summary, the checks, and the verdict — then exits with code 0 when the scenario is proven. Timestamped reports are saved under:

Reports/Session-09/Session-09-No-Bulkhead-RunReport-<timestamp>.txt
Reports/Session-09/Session-09-Bulkhead-RunReport-<timestamp>.txt

💡 No setup needed. The three dependencies and their pools all live inside the test runner. There is no server to start and no database to seed — just run the command.

Bulkhead isolation does not fix the slow dependency itself — Payment was just as slow as before. What it does is limit the blast radius: the slow work is trapped in its own compartment, and the healthy work sails through untouched. One leak should not sink the whole ship.

Part of the **Fundamentals of Distributed Systems* series — building Wassal, a distributed food-delivery lab, one concept at a time.*

Day 8 — Circuit Breaker: Stops the Retry Storm

mohamed Tayel — Sat, 25 Jul 2026 20:26:37 +0000

Retries rescue you from a blip. But when a dependency is fully down, those same retries pile on and become a storm. Today the client learns to stop knocking on a door that won't open — and to notice, on its own, when the door opens again.

🙏 Credit where it's due — This series is my attempt to internalize and share what I learned from Mahmoud Youssef's excellent course, Fundamentals of Distributed Systems on Udemy. The course material, structure, and topic flow are his work; the explanations, code examples, and diagrams in these articles are my own rewrite in my own words. If you find this useful, please consider taking the course — it goes much deeper than these articles can.

Recap — where retries left us

Day 6 taught the client to retry with backoff + jitter. Day 7 taught the server to push back with 429 + Retry-After, and the client (Part 2) to obey it — turning a temporary rejection into an eventual success.

That whole story assumes the dependency is still alive, just busy. Day 8 asks the harder question: what if the dependency is genuinely down? Now retrying isn't patience — it's a hammer.

The one idea to take away

💡 Retries help with temporary failure. A circuit breaker protects the system when failure is sustained. When every call is failing, the smartest thing a client can do is stop calling for a while — fail fast, give the dependency room to recover, and probe gently to see when it's back.

1. The Problem — retries become a storm

Take the polite retry client from Day 7 and point it at a dependency (say a payments service) that is completely down. Retry-After never arrives because the dependency isn't answering at all — it just fails. So the client does what it was told: try, wait, try again, up to its budget.

With 12 concurrent operations and 4 attempts each, that is 12 × 4 = 48 calls slammed into something that is already on the floor. Every call fails. Nothing completes. The retries didn't help — they multiplied the load on a broken dependency exactly when it needed breathing room.

🐛 The retry storm. A naive retrying client treats "down" the same way it treats "busy." Against a sustained outage that means maximum wasted load: 48 dependency calls, 0 completions. This is the trap a circuit breaker exists to stop.

2. Mental Model — the three states

A circuit breaker is a tiny state machine that sits in front of the dependency call. It watches failures and decides whether a call is even allowed to go out.

● Closed — Normal. Calls flow through. Consecutive failures are counted.
▲ Open — Tripped. Calls fail fast — the dependency isn't touched at all — for a cooldown.
◆ Half-Open — After cooldown, one probe is allowed through to test the waters.

The loop closes itself: a probe that fails sends the breaker back to Open for another cooldown; a probe that succeeds returns it to Closed and normal traffic resumes.

The cycle, as a timeline

Here is the actual path the breaker walked in our run (one shared breaker protecting the dependency, with the dependency recovering partway through):

Time	State	What happens
~0 ms	CLOSED	First calls go out. The dependency is down → they fail. After 3 consecutive failures…
~5 ms	→ OPEN	Breaker trips. The next wave of attempts fails fast — no dependency calls made.
~250 ms	→ HALF-OPEN	Cooldown elapsed → one probe allowed. Dependency still down (< 350 ms) → probe fails.
~250 ms	→ OPEN	Failed probe re-opens the breaker for another cooldown.
~500 ms	→ HALF-OPEN	Cooldown elapsed again → one probe allowed. Dependency has recovered (≥ 350 ms) → probe succeeds.
~500 ms	→ CLOSED	Breaker closes. Remaining operations sail through and complete.

💡 One breaker per dependency. All 12 operations share a single breaker, because the thing being protected is the shared dependency — not any one operation. When it trips, it trips for everyone; when it heals, it heals for everyone.

3. What does "fail fast" mean?

"Fail fast" sounds strange at first — why would failing ever be a good thing? The trick is in the word fast.

When the dependency is down, a normal call doesn't fail instantly. It waits — for a connection, for a timeout, for a retry delay — and only then gives up. Multiply that wasted waiting across 12 operations × 4 attempts and you get a pile of slow, pointless calls hammering something that is already broken.

Fail fast means the client's own circuit breaker says "no" immediately, before any network call goes out. The attempt is rejected by a tiny in-memory guard in microseconds, instead of waiting on a dependency that is almost certainly still down.

💡 🏠 Everyday picture. Imagine knocking on a shop door and seeing a "CLOSED" sign in the window. You don't stand there knocking for 30 seconds hoping — you read the sign and move on in a split second. The "Open" circuit breaker is that CLOSED sign: it lets the client give up instantly instead of waiting.

In our run, 22 attempts failed fast while the circuit was Open. Those 22 attempts never touched the payments dependency at all — that is exactly the load the breaker shed.

✅ Why this helps the dependency. Every fast-failed attempt is one less call landing on a service that is trying to restart. Failing fast isn't giving up on the user — it's giving the broken dependency the quiet it needs to actually come back.

4. Why Half-Open exists

If the breaker only had Closed and Open, we'd hit a new problem: once it tripped Open, how would it ever know the dependency came back? If it stays Open forever, it has just turned a temporary outage into a permanent one — now we are the reason nothing works.

But the opposite is also dangerous. If, the moment the cooldown ends, the breaker flung all traffic back at the dependency at once, it could knock a still-fragile service straight back down. We need a gentle, careful test — not a stampede.

Half-Open is that careful test. After the cooldown (OpenDurationMs = 200), the breaker lets exactly one probe request through while everyone else keeps fast-failing:

If the single probe succeeds → the dependency is back → the breaker closes and normal traffic resumes for everyone.
If the probe fails → still down → straight back to Open for another cooldown, and we try again later.

💡 🚪 Everyday picture. After the shop's posted closing time, you don't send the whole queue crashing through the door. You send one person to gently try the handle. If it opens, everyone follows. If it's still locked, the queue waits a bit longer. That one careful tester is the Half-Open probe.

In our run this happened twice: the first probe (at ~250 ms) landed while the dependency was still down and re-opened the breaker; a later probe (after the 350 ms recovery) succeeded and closed it. That is the self-healing loop working on its own.

5. The Knobs — and why the timing is what it is

The breaker is hand-written with four visible constants:

Constant	Meaning
`MaxAttempts = 4`	Retry budget per operation (same as the BEFORE scenario, so the two compare fairly).
`RetryDelayMs = 250`	Wait between attempts.
`FailureThreshold = 3`	Consecutive failures that trip the breaker Closed → Open.
`OpenDurationMs = 200`	Cooldown the breaker stays Open before allowing a Half-Open probe.

The simulated dependency is down while elapsed < RecoveryAfterMs = 350, then it recovers.

Why `OpenDurationMs` and `RecoveryAfterMs` were tuned (a teaching choice, not a fudge)

One operation's lifetime is bounded by its retry budget:

operation lifetime ≈ (MaxAttempts − 1) × RetryDelayMs
                   = (4 − 1) × 250 ms
                   = ~750 ms

That ~750 ms ceiling is the whole reason the other two numbers are what they are. For this lesson to actually show a full recovery, the breaker must travel Open → Half-Open → (failed probe) → Open → Half-Open → (successful probe) → Closed before the operations run out of attempts and finish.

If OpenDurationMs were large (e.g. the 750 ms a real system might use), the cooldown alone would consume an operation's entire lifetime. The operations would all finish while the circuit was still Open — you'd never see a Half-Open probe, never see the circuit close, and never see recovery. The lesson would be invisible.
So OpenDurationMs = 200 (shorter than 750 ms) lets the breaker cycle through Half-Open at least twice inside the budget.
RecoveryAfterMs = 350 is placed so the first probe lands while the dependency is still down (proving fast-fail and re-open), and a later probe lands after recovery (proving the close). It sits comfortably below the ~750 ms ceiling, leaving room for completed work after the breaker closes.

⚠️ In production these numbers are bigger. A real breaker might stay Open for many seconds and probe slowly. We shrank the clock here purely so a single short, deterministic run demonstrates the full Closed → Open → Half-Open → Closed cycle. The mechanism is identical; only the dial settings are scaled down for teaching.

6. How the Code Works

Let's walk through the AFTER scenario — Session08CircuitBreakerScenario.cs — one piece at a time. No huge code dumps; just the moving parts and how they fit together. Everything below uses the actual names from the file.

① The constants — the dials of the experiment

Constant	Meaning
`OperationCount`	How many logical operations fire at once. It comes from the session context (`ctx.OperationCount`) and is 12 in this run.
`MaxAttempts = 4`	Each operation may try up to 4 times before it gives up.
`RetryDelayMs = 250`	The wait between two attempts of the same operation.
`FailureThreshold = 3`	Three consecutive failures while Closed trip the breaker to Open.
`OpenDurationMs = 200`	How long the breaker stays Open (fast-failing) before it lets one probe through.
`RecoveryAfterMs = 350`	The simulated dependency is down before this time and healthy after it.

② The fake dependency — no server, no database

The dependency is not a real payments service. It is a tiny local method, CallDependencyAsync(), living inside the test runner. It does only two things: count itself, and answer "am I up yet?" purely from the clock.

async Task<bool> CallDependencyAsync()
{
    Interlocked.Increment(ref dependencyCallsMade);   // count every real call
    await Task.Yield();
    return clock.ElapsedMilliseconds >= RecoveryAfterMs; // down < 350 ms, up after
}

💡 In-process only. While elapsed time is less than RecoveryAfterMs it returns false (down); once elapsed reaches RecoveryAfterMs it returns true (healthy). There is no network call, no real server, and no database anywhere in this lesson — just a stopwatch and a counter.

③ The three breaker states

The breaker is the small CircuitBreaker class, and its mood is a single enum BreakerState { Closed, Open, HalfOpen }:

● Closed — Healthy. Every call is allowed through, and consecutive failures are counted.
▲ Open — Tripped. Calls are rejected instantly (fast fail) — the dependency is not touched — for OpenDurationMs.
◆ HalfOpen — Testing the water. Exactly one probe is let through; everyone else still fast-fails until that probe resolves.

④ The gate method — ask before you call

The key idea of the whole lesson lives in one method, TryAcquire().

✅ Before calling the dependency, the operation asks the breaker first.

TryAcquire() returns an AcquireResult(bool Allowed, bool IsProbe). The operation then reacts to that answer:

var acq = breaker.TryAcquire();
if (!acq.Allowed)
{
    fastFails++;                 // breaker said NO → fast fail, dependency untouched
    // …wait, then try the next attempt
    continue;
}

depCalls++;                      // breaker said YES → the call reaches the dependency
var ok = await CallDependencyAsync();

If the breaker says yes (Allowed = true) the call actually reaches CallDependencyAsync().
If the breaker says no (Allowed = false) the attempt becomes a fast fail — fastFails is incremented and the dependency is never touched.
When Open and the cooldown has elapsed, TryAcquire() promotes exactly one caller to a probe (IsProbe = true) and flips the state to HalfOpen.

⑤ Recording failures

When a call fails, the operation tells the breaker with RecordFailure(acq.IsProbe). Inside, _consecutiveFailures climbs:

While Closed, reaching FailureThreshold = 3 consecutive failures trips the breaker to Open and records the moment. Three is the line in the sand: a blip or two is tolerated, but a sustained run of failures means "stop hammering."
If a Half-Open probe fails (wasProbe = true), the breaker goes straight back to Open for another full cooldown — no need to wait for three more failures; one failed probe is proof the dependency is still down.

⑥ Recording success

A successful call calls RecordSuccess(acq.IsProbe), which:

Resets _consecutiveFailures back to 0 and sets the state to Closed.
If the success was a Half-Open probe, it also flips ClosedAfterRecovery = true — that is the breaker noticing, on its own, that the dependency came back. A good probe closes the circuit and normal traffic resumes.

⑦ The retry loop — every attempt asks first

Each operation runs the same loop up to MaxAttempts = 4 times. The crucial detail: every attempt begins by asking the breaker, so the same attempt can become either a real dependency call or a fast fail depending on the breaker's current state.

One attempt, step by step:

breaker.TryAcquire() — ask permission.
- Not allowed? fastFails++, wait RetryDelayMs, go to next attempt. Dependency untouched.
- Allowed? depCalls++, call CallDependencyAsync().
  - Success → RecordSuccess → mark completed → break (stop early, no more attempts).
  - Failure → RecordFailure → wait RetryDelayMs → next attempt.

So early attempts (breaker Closed) tend to be real dependency calls; the wave right after the breaker trips becomes fast fails; later attempts may be probes. That mix is exactly what the report counts.

⑧ Reading the report numbers

Two independent counters drive the verdict: dependencyCallsMade (incremented inside the dependency) and fastFails (incremented when the gate says no). Here is how the headline numbers fall out:

BEFORE = 48 dependency calls. With no breaker, all 12 operations always burn their full 4 attempts and every attempt hits the dependency: 12 × 4 = 48.
AFTER = 25 dependency calls + 22 fast fails. With the breaker in front, only 25 attempts actually reached the dependency; the other 22 were rejected instantly while the circuit was Open. Those 22 calls are the load the breaker shed.
Why total attempts can be 47, not 48. Every attempt is either a dependency call or a fast fail, so total attempts = 25 + 22 = 47. The baseline is 48 (12 × 4). The "missing" attempt is simply an operation that completed early: once a call succeeds, the loop breaks and the operation does not use its remaining attempt(s).
This is not a bug. A lower-than-baseline total attempt count is the breaker working as designed — finishing the moment the dependency is healthy instead of mechanically spending all four attempts. Fewer calls and fewer wasted attempts is the win.

💡 Quick check: dependencyCallsMade (25) + fastFails (22) = totalAttempts (47) ≤ baseline (48). If you see 48, no operation finished early; if you see less, at least one did.

7. BEFORE — No Circuit Breaker

File: tools/Wassal.SessionTests/Sessions/Session08NoCircuitBreakerScenario.cs

Command:

dotnet run --project tools/Wassal.SessionTests -- session-08-no-circuit-breaker

What it does: fires 12 concurrent operations at a dependency that is always down. Each operation retries naively up to MaxAttempts — no breaker, nothing to stop the calls.

Verified result: 12 operations, each burning all 4 attempts → 48 dependency calls, 0 completed, 12 failed, average attempts/operation = 4.00.

VERDICT: NO CIRCUIT BREAKER RETRY STORM DEMONSTRATED ✅

🐛 Passes by proving the problem. Like every BEFORE diagnostic, success here means the bug was demonstrated: every operation consumed its full retry budget against a down dependency, and all 48 calls were wasted.

8. AFTER — With the Circuit Breaker

File: tools/Wassal.SessionTests/Sessions/Session08CircuitBreakerScenario.cs

Commands (the short session-08 is an alias for the AFTER scenario):

dotnet run --project tools/Wassal.SessionTests -- session-08-circuit-breaker

# Short alias → runs the AFTER (circuit-breaker) scenario:
dotnet run --project tools/Wassal.SessionTests -- session-08

What it does: the same 12 concurrent operations, but every attempt goes through a shared circuit breaker. The dependency is down for the first 350 ms, then recovers. The breaker trips, fast-fails the open-circuit attempts, probes at each cooldown, and closes once the dependency is back.

Verified result:

12 operations · 12 completed · 0 failed
Circuit opened: 2 · Half-open probes: 2 · Closed after recovery: YES · final state: Closed
Dependency calls made: 25 (vs the 48 baseline) → 23 calls avoided
Fast-failed attempts: 22 (rejected instantly while the circuit was Open — dependency untouched)

VERDICT: CIRCUIT BREAKER PROTECTED THE DEPENDENCY ✅

✅ The proof is protection + recovery, not perfect completion. The verdict does not require all 12 to finish — under concurrency some operations can exhaust their attempts before recovery. What it requires is that the breaker opened, fast-failed, made fewer calls than the baseline, allowed a half-open probe, closed after recovery, and completed at least one operation. In this run all 12 happened to complete — even better.

BEFORE vs AFTER

Metric	BEFORE — no breaker	AFTER — circuit breaker	Difference
Dependency calls	48	25	23 avoided
Completed operations	0	12	+12
Fast-failed attempts	0	22	load shed
Auto-recovery	none	half-open → closed	self-healing
Verdict	RETRY STORM DEMONSTRATED ✅	DEPENDENCY PROTECTED ✅

Same outage, same retry budget, same 12 operations. The only change is a small state machine in front of the call — and it nearly halved the load on the broken dependency (48 → 25) while taking completions from 0 to 12.

9. What We Proved · What We Didn't Do

✅ What we proved:

Against a sustained outage, blind retries make 48 wasted calls.
A circuit breaker trips Open after a few failures and fails fast — sparing the dependency.
It probes in Half-Open and re-opens if still broken.
It closes automatically when a probe succeeds after recovery.
Net effect: 48 → 25 calls (23 avoided) and 0 → 12 completed.

🚧 What we intentionally did not do:

No Polly — the breaker is hand-written so the states are visible.
No external package or resilience library.
No real server dependency — the dependency is simulated in-process.
No database involved in this lesson.
No production-ready implementation (no sliding-window stats, half-open concurrency limits, metrics, or per-endpoint breakers).

10. The Important Lesson — three tools, three different jobs

Days 6, 7 and 8 each added a resilience tool. Beginners often blur them together, but they solve different problems and live on different sides of the wire. Here is the one-line difference:

🔁 Retry — "try harder" — Lives on the client. Assumes the failure is a brief blip, so it tries again (with backoff + jitter). Great for transient hiccups — but against a real outage it just tries harder at the wrong moment.
🚦 Rate limiting — "protect the server from too much traffic" — Lives on the server. The server defends itself from overload by rejecting excess requests (429 + Retry-After). It protects the thing being called from the callers.
⛓ Circuit breaker — "protect a failing dependency from being hammered" — Lives on the client. When a dependency is clearly down, the client stops calling it (fail fast) so the broken dependency gets room to recover — then probes gently to detect when it's back.

💡 The mental shortcut: Retry says "keep trying." Rate limiting says "the server protects itself." Circuit breaker says "the caller protects the callee — by knowing when to stop." Retry and circuit breaker are partners: retry handles blips, the breaker steps in when blips turn into an outage.

11. Run It Yourself

① Run the AFTER scenario (the circuit breaker):

dotnet run --project tools/Wassal.SessionTests -- session-08-circuit-breaker

② Compare against the BEFORE scenario (no breaker):

dotnet run --project tools/Wassal.SessionTests -- session-08-no-circuit-breaker

The run prints the per-operation table, the breaker summary, the PASS/FAIL checks, and the final verdict — then exits with code 0 when the proof holds.

③ Where to find the report:

Every run also saves a timestamped text report under the Reports/Session-08/ folder:

Reports/Session-08/Session-08-Circuit-Breaker-RunReport-<timestamp>.txt

The run behind this article is saved as Reports/Session-08/Session-08-Circuit-Breaker-RunReport-2026-06-28-130200.txt.

💡 No setup needed. The dependency, the clock, and the breaker all live inside the test runner. There is no server to start and no database to seed — just run the command and read the report.

Conclusion

A circuit breaker is one of the smallest pieces of code with one of the biggest payoffs in distributed systems. In this lesson, a single hand-written state machine in front of the dependency call turned a wasteful storm — 48 calls, 0 completions — into protected, self-healing traffic: 25 calls, 22 fast fails, 12 completions, and an automatic recovery. The client stopped knocking on a locked door, waited, gently checked the handle, and walked through the moment it opened.

💡 Retries help with temporary failure. Circuit breakers protect the system when failure is sustained. Day 6 made retries gentle. Day 7 made the server push back. Day 8 teaches the client when to stop pushing — and how to notice, by itself, that the dependency has come back.

Part of the **Fundamentals of Distributed Systems* series — building Wassal, a distributed food-delivery lab, one concept at a time.*

Distributed Systems in .NET — Day 7 — Part 2

mohamed Tayel — Sat, 25 Jul 2026 20:26:32 +0000

The server already learned to say "not now." Now the client learns to listen.

⚠️ This is Day 7 — Part 2, not Lesson 8. It is a direct continuation of the 429 + Retry-After lesson from Part 1. Part 1 was the server side (rate limiting); Part 2 is the client side (cooperating with the server's signal). Same lesson, second half.

Recap from Day 7 · Part 1

In Part 1 we taught the server to defend itself with a token-bucket rate limiter on POST /api/orders:

When the bucket runs dry, the server returns HTTP 429 Too Many Requests with a Retry-After header.
Rejected requests did no database work — a 429 creates no order, so there is never a half-finished mess to clean up.
The server stayed healthy under a flood by shedding the excess instead of trying to serve everything.

The Part 1 numbers, side by side:

Run	Requests	Accepted (200)	Rejected (429)	Retry-After	DB rows
BEFORE (no limiter)	30	30	0	—	30
AFTER (token bucket)	30	5	25	25 / 25	5

That AFTER row is exactly where Part 2 begins. The server protected itself — but 25 requests walked away with a 429. What happens to those 25? That depends entirely on the client.

💡 A note on the numbers. For the Part 2 demo we use a smaller, faster run — 12 logical operations instead of 30 — so the retry behaviour is easier to read in the console. The lesson is the same: some requests hit 429; the naive client loses them, while the polite client waits and completes them.

The one idea to take away

💡 429 is not "never." 429 is "not now." The server already did its job in Part 1. In Part 2 the client does its job: read Retry-After, wait, and retry the same operation with the same key — until it completes.

Skills you'll lean on again (Lessons 3–7)

Idempotency-Key = one key per business operation
Idempotency makes a retry safe (no duplicates)
Backoff + jitter to avoid a stampede (Lesson 6)
429 + Retry-After from the server (Part 1)
Before/After discipline + saved reports

Skills you'll add in Part 2

Reading the Retry-After header on the client
Waiting at least that long before retrying
Retrying the SAME operation with the SAME key
Bounding attempts so the client never loops forever
Turning a 429 into an eventual success

Part 2 Goal

Take the Part-1 server exactly as it is (no server change) and put two clients in front of it. First a naive client that treats a 429 as a final failure — and loses operations. Then a polite client that reads Retry-After, waits, and retries the same operation with the same Idempotency-Key — until every operation completes, with no duplicates.

The Problem — a naive client throws work away

A naive client sees a non-200 status and gives up. To it, 429 looks like "failed," so it reports failure to the user and stops. But the server didn't say the operation was impossible — it said "not right now."

12 logical operations, fired at a rate-limited server.

bucket allows ~5 now → 5 operations get 200 OK  → completed
the other ~7 hit the empty bucket → 429          → naive client GIVES UP

Result with a naive client:
  completed = 5
  lost      = 7   ← thrown away, never retried
  DB rows   = 5   ← only the accepted operations created orders

🐛 The loss is silent. The data that does exist is correct (5 orders for 5 accepted requests), so nothing looks broken on the server. But 7 real customer intents just vanished because the client mistook "not now" for "never." DB rows match only the accepted requests — not what the user actually asked for.

Mental Model — 429 means "not now," not "never"

403 Forbidden        → "No. You are not allowed." (never)
404 Not Found        → "That doesn't exist."       (never, here)
429 Too Many Requests→ "Not now — try again soon." (later!)

Retry-After: 1       → "Here is exactly how long to wait: ~1 second."

This is the key reframe. A 429 is a temporary answer with an expiry date attached. The server is not rejecting the operation forever — it is asking the client to come back. And it even tells the client when via Retry-After. A good client treats that as an instruction, not an error.

السيرفر مش بيقولك مستحيل. هو بيقولك: استنى شوية وجرب تاني.

(The server isn't telling you "impossible." It's telling you: wait a little and try again.)

The Correct Client Behavior

For each logical operation, the polite client follows this loop:

Generate one Idempotency-Key for the operation (once, up front).
Send the request.
If 200 OK → done.
If 429 →
- read Retry-After
- wait Retry-After seconds
- optionally add a small jitter on top (so clients don't all return on the same tick)
- retry using the SAME Idempotency-Key
Stop after a max attempt count (never loop forever).

In pseudocode (illustrative — the real loop lives in the verification scenario, not the app):

// One key for the whole operation — generated ONCE, reused on every retry.
var idempotencyKey = Guid.NewGuid().ToString();
const int maxAttempts = 5;

for (var attempt = 1; attempt <= maxAttempts; attempt++)
{
    var response = await PostOrderAsync(body, idempotencyKey);

    if (response.Status == 200) return response;        // ✅ completed

    if (response.Status == 429)
    {
        var retryAfter = response.RetryAfterSeconds ?? 1; // server told us when
        var jitter     = Random.NextDouble() * 0.25;      // small wobble (Lesson 6)
        await Task.Delay(TimeSpan.FromSeconds(retryAfter + jitter));
        continue;                                         // try the SAME key again
    }

    throw new Exception($"Unexpected status {response.Status}");
}
// Bounded: after maxAttempts, give up gracefully (don't hammer forever).

⚠️ Wait at least Retry-After — and add jitter on top. Waiting less than Retry-After disobeys the server and just earns another 429. Waiting the exact same value on every client re-creates the thundering herd from Lesson 6 — they all come back on the same tick. So: honor Retry-After as the floor, then add a little randomness.

Why the SAME Idempotency-Key matters

The Idempotency-Key belongs to the operation, not the attempt. Generate it once, reuse it on every retry of that operation.

✅ CORRECT — same key across retries

operation "create my order"  →  key = ABC-123  (generated once)
   attempt 1 → 429 (not now)
   attempt 2 → POST  Idempotency-Key: ABC-123   ← same key
   attempt 3 → 200 OK → Order #55
Result: exactly 1 order. ✅

❌ WRONG — a new key per retry

operation "create my order"
   attempt 1 → key KEY-1 → 429
   attempt 2 → key KEY-2 → 200 → Order #55
   attempt 3 → key KEY-3 → 200 → Order #56
Result: the server sees 3 DIFFERENT operations → duplicate orders. 💥

Because the key is stable, the server recognizes a retry as the same intent and never creates a second order. This is the guarantee built in Lessons 3–5 (in-memory dedup → durable store → concurrency-safe replay). Part 2 simply depends on it: idempotency is what makes "just retry" a safe thing to do.

💡 Rule of thumb. New key = new intent. Same retry = same key. If you can't point to where the key was created once and held across retries, your "safe retry" isn't safe.

Verification Discipline — same rule, Part 2

As always, we prove this with repeatable .NET scenarios, not throwaway PowerShell. Part 2 adds two scenarios that run against the unchanged Part-1 rate-limited server — only the client behaviour differs. Each cleans the DB, fires the operations, prints a readable table, applies a clear verdict, and saves a timestamped report that previous runs never overwrite.

The naive scenario expects loss (operations thrown away). The polite scenario expects full completion (every operation eventually succeeds, no duplicates). The only thing that changes is whether the client obeys Retry-After.

Two scenarios with different verdict logic

Both fire the same 12 operations at the same rate-limited server. What differs is how the client reacts to a 429:

① BEFORE — Naive gives up (a diagnostic; happy when loss is demonstrated)

dotnet run --project tools/Wassal.SessionTests -- session-07-part-2-naive-gives-up

Verdict — PASS when:

At least one 429 observed
completed > 0 and lost > 0
DB rows == completed (lost ops created nothing)

② AFTER — Polite obeys Retry-After (a proof; happy only when everything completes, exactly once)

dotnet run --project tools/Wassal.SessionTests -- session-07-part-2-polite-retry-after

# Short alias:
dotnet run --project tools/Wassal.SessionTests -- session-07-part-2

Verdict — PASS when:

completed == total operations
DB rows == total, unique ids == total
every retry waited ≥ Retry-After; attempts > operations

⚠️ Same input, opposite verdicts — on purpose. Both keep the database correct (no duplicates). The naive client just leaves work unfinished; the polite client finishes it. The improvement Part 2 proves is completion, not correctness.

BEFORE — Watch the Naive Client Lose Operations

Start the Part-1 server (token bucket already in place — no change):

dotnet run --project src/Wassal.Monolith --launch-profile https

Then run the naive scenario:

dotnet run --project tools/Wassal.SessionTests -- session-07-part-2-naive-gives-up

What it does: fires 12 logical operations (each its own Idempotency-Key), one attempt each. A 200 is completed; a 429 is lost-429 (no retry).

Expected shape:

12 logical operations
some complete (≈5), some get 429 (≈7)
lost > 0
DB rows == completed only

🐛 Diagnostic PASS = loss observed. Like every BEFORE scenario, this one "passes" when it proves the problem: at least one 429, some completed, some lost, and the DB holding only the completed operations. That is the naive client throwing work away.

✅ Done when: lost > 0 and DB rows == completed (the lost operations created nothing).

AFTER — The Polite Client Completes Everything

Same server, same 12 operations — now with a client that listens:

dotnet run --project tools/Wassal.SessionTests -- session-07-part-2-polite-retry-after

# Short alias:
dotnet run --project tools/Wassal.SessionTests -- session-07-part-2

What it does: for each operation, on a 429 the client reads Retry-After, waits that long (plus a little jitter), and retries with the same Idempotency-Key — up to a bounded max attempts.

Expected shape:

12 logical operations
429s may happen as intermediate attempts (not final)
the client waits according to Retry-After
the same Idempotency-Key is reused per operation
all operations eventually complete
DB rows == total operations
unique order ids == total operations
attempts > operations (retries actually happened)

✅ Loss → zero. The same flood that left 7 operations behind in the naive run now completes all 12. Idempotency keeps it to exactly 12 orders (no duplicates), and obeying Retry-After keeps the client from hammering the recovering server. The 429s become speed bumps, not dead ends.

✅ Done when: completed == 12, DB rows == 12, unique ids == 12, and attempts > 12.

Console & Report Expectations

Each scenario prints a per-operation table (operation number, short key, status, result, Retry-After seen, attempts, elapsed ms, order id) plus a summary block and a verdict. Reports save automatically — timestamped, never overwritten — under:

Reports/Session-07-Part-2/

Example file names:

Session-07-Part-2-Naive-Gives-Up-RunReport-yyyy-MM-dd-HHmmss.txt
Session-07-Part-2-Polite-Retry-After-RunReport-yyyy-MM-dd-HHmmss.txt

💡 Why save reports? Same discipline as every session: a saved, timestamped report is evidence you can attach to this writeup and compare run-to-run. The naive report shows loss; the polite report shows full completion — side by side they tell the whole Part-2 story.

Implementation Walkthrough — What We Built & Tested

A beginner-friendly tour of the two verification scenarios behind Day 7 Part 2 — the naive client that loses work, and the polite client that obeys Retry-After and completes everything exactly once. The server (the Part 1 token-bucket rate limiter) was not changed — Part 2 is about client behaviour only.

The whole idea in one line: Part 1 taught the server to say "not now" with HTTP 429 + Retry-After. Part 2 is about the client: does it listen? We built two clients and ran them against the same rate-limited server to find out.

Naive client — treats 429 as a final failure and throws the work away.
Polite client — reads Retry-After, waits, adds a little jitter, and retries the same operation with the same Idempotency-Key until it completes.

BEFORE — The Naive Client Gives Up

File: tools/Wassal.SessionTests/Sessions/Session07Part2NaiveGivesUpScenario.cs

Command:

dotnet run --project tools\Wassal.SessionTests -- session-07-part-2-naive-gives-up

What it does:

Sends 12 logical operations concurrently at the rate-limited POST /api/orders.
Each operation has its own Idempotency-Key (distinct operations, not retries of one).
Each operation makes only one attempt.
If the server returns 200 → the operation is completed.
If the server returns 429 → the operation is marked lost-429.
No retry happens. The naive client gives up.

🐛 Why this "passes" by failing. This is a BEFORE diagnostic. It is designed to prove the problem exists. PASS here means the bug was demonstrated: the naive client mistook 429 ("not now") for a permanent failure and lost real customer work.

Verified result:

Metric	Value	Meaning
Total operations	12	Everything we fired
Completed (200)	5	Got through the bucket
Lost (429, gave up)	7	Thrown away — never retried
DB rows (Orders)	5	== completed only
Unique order ids	5	No duplicates, but work is missing

VERDICT: NAIVE LOSS DEMONSTRATED ✅ (diagnostic)

The data that exists is correct — DB rows == completed — so nothing looks broken on the server. But 7 real intents silently vanished. That silent loss is exactly what the polite client fixes. (The 5/7 split can vary slightly with timing; the verdict only requires some loss, not exactly 5 and 7.)

AFTER — The Polite Client Obeys Retry-After

File: tools/Wassal.SessionTests/Sessions/Session07Part2PoliteRetryAfterScenario.cs

Command:

dotnet run --project tools\Wassal.SessionTests -- session-07-part-2-polite-retry-after

Short alias (runs the same polite scenario):

dotnet run --project tools\Wassal.SessionTests -- session-07-part-2

What it does:

Sends the same 12 logical operations concurrently at the same Part 1 server.
Each operation gets one Idempotency-Key generated once, up front.
If the server returns 200 → the operation is completed.
If the server returns 429, the client:
- reads Retry-After,
- waits at least Retry-After seconds,
- adds a small jitter on top,
- retries using the SAME Idempotency-Key.

The knobs (explicit and visible in code):

Bounded retries: MaxAttempts = 6 — the loop can never run forever.
Jitter: 0..250 ms added on top of Retry-After (never less than it).
Fallback: if Retry-After is missing or unparsable → use 1 second.
Never generate a new key per retry — a retry is the same intent.

Verified result:

Metric	Value	Meaning
Total operations	12	Same as the naive run
Completed (200)	12	Every operation finished
429 intermediate attempts	9	Speed bumps, not dead ends
Total attempts (POSTs)	21	Retries actually happened
DB rows (Orders)	12	== total operations
Unique order ids	12	No duplicates
Every retry honored Retry-After	YES	The client obeyed the server

VERDICT: PASS ✅

✅ 21 attempts → only 12 orders. The client made 21 HTTP attempts, but the database holds only 12 orders with 12 unique ids. That gap is the proof of correctness: every retry reused the same Idempotency-Key, so the server recognized the retry as the same operation and never created a duplicate. Completion went up; duplicates stayed at zero.

💡 Same flood, opposite outcome. The naive run left 7 operations behind. The polite run — same 12 operations, same rate limiter — completed all 12 by turning each 429 into a short wait and a retry.

How to Run It Locally

You need two terminal windows: one keeps the server running, the other runs the scenarios.

Window 1 — start the server (keep this open):

cd /d D:\books\distributed-system\Wassal\src
dotnet run --project Wassal.Monolith --launch-profile https

Wait until you see this line — it means the server is ready:

Now listening on: https://localhost:7126

⚠️ Leave Window 1 open. The server must stay running the whole time you run the scenarios in Window 2. If you close it (or press Ctrl+C), every scenario will fail to connect to https://localhost:7126.

Window 2 — build, then run the scenarios:

cd /d D:\books\distributed-system\Wassal

dotnet build

REM BEFORE — naive client (should show loss)
dotnet run --project tools\Wassal.SessionTests -- session-07-part-2-naive-gives-up

REM AFTER — polite client (should complete everything)
dotnet run --project tools\Wassal.SessionTests -- session-07-part-2-polite-retry-after

REM AFTER via the short alias (same polite scenario)
dotnet run --project tools\Wassal.SessionTests -- session-07-part-2

REM Part 1 regression — the server should still shed load correctly
dotnet run --project tools\Wassal.SessionTests -- session-07-rate-limited

💡 What to expect. The naive run prints NAIVE LOSS DEMONSTRATED ✅, the polite run and its alias print PASS ✅, and the Part 1 regression still prints PASS ✅. Each run also saves a timestamped report.

What I Should Understand Before Moving On

429 means "not now", not "never" — a temporary answer with an expiry attached; come back soon, the door isn't locked forever.
Retry-After tells you how long to wait — the server hands the client an exact instruction. Treat it as a rule, not noise.
Naive client ignores Retry-After — it treats 429 as final and loses real work; completion drops, silently.
Polite client obeys Retry-After — it waits, retries, and completes every operation. 429 becomes a speed bump.
The key belongs to the operation — Idempotency-Key identifies the logical operation, not a single HTTP attempt.
Same retry = same key · New key = new intent — reuse the key on retries so the server sees one operation. A new key means a new order.
Bounded retries prevent infinite loops — MaxAttempts = 6 guarantees the client eventually stops instead of hammering forever.
Jitter de-synchronizes clients — a small random add-on stops every client from retrying on the exact same tick (no new stampede).
DB rows are the proof — naive — DB rows = completed only. The lost operations created nothing.
DB rows are the proof — polite — DB rows = total operations, with unique ids = total operations.

الخلاصة بالمصري 🇪🇬

السيرفر لما يرجع 429 هو مش بيقول إن العملية فشلت للأبد. هو بيقول للعميل: استنى شوية وجرب تاني. العميل الغلط بيسيب الطلب يضيع. العميل الصح بيستنى Retry-After ويرجع يحاول بنفس Idempotency-Key، علشان السيرفر يفهم إن دي نفس العملية مش طلب جديد. عشان كده كل العمليات كملت، والداتابيز فيها 12 أوردر فقط من غير أي تكرار.

Final Comparison — Naive vs Polite

Scenario	Behavior on 429	Completed	Lost	Attempts	DB Rows	Verdict
Naive	Gives up	5	7	12	5	NAIVE LOSS DEMONSTRATED ✅
Polite	Waits + retries same key	12	0	21	12	PASS ✅

Read the two rows together: the polite client made more attempts (21 vs 12) but produced the same kind of clean database — no duplicates — while finishing every operation (12 vs 5). More attempts, zero duplicates, full completion. That is the whole lesson in one table.

Final takeaway — the whole contract

💡 Backoff is the client choosing to slow down (Lesson 6). Rate limiting is the server enforcing the slowdown (Part 1). 429 + Retry-After is the contract between them.

The polite client completes the operation safely by listening to the server — waiting Retry-After — and reusing the same Idempotency-Key so the eventual retry creates exactly one order. Server protection (Part 1) + client cooperation (Part 2) = every operation completes, exactly once, even through a rate limiter.

Part 1 gave the server a voice (429 + Retry-After). Part 2 gave that voice a listener. Together they complete the rate-limiting lesson. This is not Lesson 8 — it's the second half of Day 7. Lesson 8 will move on to the next topic.

Part of the **Fundamentals of Distributed Systems* series — building Wassal, a distributed food-delivery lab, one concept at a time.*

Distributed Systems in .NET — Day 7

mohamed Tayel — Sat, 25 Jul 2026 20:25:54 +0000

Lesson 6 taught the client to back off — but you can't trust every client to. Today the server learns to defend itself: shed the excess, and tell callers exactly when to come back.

Bots, buggy SDKs, and panicking apps will still hammer you. In this session we flood the order endpoint and watch the unprotected server accept every single request — no backpressure at all. Then we add a token-bucket rate limiter using .NET 8's built-in middleware, so the excess gets 429 + Retry-After and the load is shed. This is Part 1 (the server side); Part 2 is the client side.

📘 Where we left off in Lesson 6: You built the client's half of idempotency: a bounded retry loop with exponential backoff + jitter, reusing one Idempotency-Key per operation. Naive retries piled into one window (a storm); polite retries spread out over time. Tagged as lesson-06-after. But Lesson 6 ended on an uncomfortable admission: "you can't trust every client to back off." A polite client cooperates — a bot doesn't. So the server cannot rely on good manners. It needs its own defence, one it enforces no matter who is calling. That defence is rate limiting, and the way it cooperates with polite clients is 429 + Retry-After.

🎯 The one idea to take away: Backoff is the client choosing to slow down. Rate limiting is the server enforcing it. You cannot outsource load control to clients. The server must have backpressure of its own — and 429 + Retry-After is how the two sides cooperate. Same theme as Lesson 6, now from the other side of the wire: idempotency keeps it correct; rate limiting keeps it standing.

What you'll reinforce and add

🔁 Reinforce — from Lessons 5 & 6:

Idempotency keeps accepted work correct (no duplicates)
Bursts of concurrent requests from the .NET runner
DB row count as the authoritative check
Before/After discipline + saved reports
Reading a console table + verdict

✨ New — added today:

Backpressure & why client politeness can't be trusted
The token bucket algorithm (rate + burst)
.NET 8's built-in rate limiter middleware
429 Too Many Requests + Retry-After
Load shedding: protect the majority by failing the excess fast

🎯 Today's goal: Flood the order endpoint with a burst of requests and watch the unprotected server accept every single one — no backpressure at all. Then add a token-bucket rate limiter using .NET 8's built-in middleware, so the excess gets 429 + Retry-After and the load is shed. Prove that the accepted requests still each create exactly one order (rejected requests do no work) — and see how the Retry-After header is exactly the signal a Lesson-6 client backs off on.

🧠 Mental Model 1 — Why client politeness isn't enough

Lesson 6 made our client polite. But your server doesn't only talk to your client. It talks to whatever shows up:

Good mobile app   →  retries with backoff + jitter   (polite, Lesson 6)
Old cached build  →  retries instantly, no backoff    (storm)
Third-party SDK   →  retries 50× in a tight loop       (worse storm)
A bot / scraper   →  10,000 requests/sec, doesn't care (attack)

The server cannot CHOOSE its callers. Politeness is voluntary.
So load control cannot live only on the client — the server needs
its OWN limit that it enforces on everyone, no exceptions.

That server-side limit is called backpressure: the system's ability to say "I am full, slow down" instead of trying to accept everything and collapsing. Without backpressure, one rude caller can consume all the capacity and starve every well-behaved user. Idempotency won't save you here — it keeps the data correct, but it happily does the (correct) work for all 10,000 requests. Correct, and on fire.

💡 Fail fast beats slow death. When overloaded, a server that rejects the excess in microseconds stays healthy for everyone it does serve. A server that tries to serve everyone slows to a crawl and fails for everyone. Rejecting some requests on purpose to protect the rest is called load shedding — and it is a feature, not a failure.

🧠 Mental Model 2 — The token bucket & the 429 + Retry-After contract

The classic way to enforce "X requests per second, with a little burst room" is a token bucket. Picture a bucket that holds a few tokens. Every request must take one token to be served. The bucket refills at a steady rate:

Bucket capacity = 5 tokens      (max burst it will absorb at once)
Refill          = 5 tokens / second  (steady allowed rate)

request arrives → is there a token?
   YES → take one, serve it (200 OK)
   NO  → bucket empty → REJECT with 429 Too Many Requests
                        + Retry-After: 1   ("come back in ~1s")

A burst of 30 at once → ~5 served now, ~25 rejected with 429.
The bucket refills, so a polite client that waits gets served next round.

The teaching analogy: the bucket is a doorman holding 5 entry tickets. Five people walk straight in; the rest are told "no tickets right now — come back in a second" (that's the Retry-After). The doorman prints 5 new tickets every second. Nobody is hurt, and the room never overflows.

⚠️ 429 + Retry-After is a contract, not just an error. A bare rejection says "go away." 429 + Retry-After: 1 says "go away for one second, then you're welcome." That second value is precisely what a Lesson-6 client feeds into its backoff: instead of guessing how long to wait, it obeys the number the server gave it. Server and client cooperate — one sets the pace, the other respects it.

🧪 Verification Discipline — same rule, new session

As always: no throwaway PowerShell. Session 7 gets its own repeatable scenarios under tools/Wassal.SessionTests, sharing the same engine and reporting as Sessions 5 & 6. Each scenario cleans the DB, fires the burst in a repeatable way, prints a readable table (including the Retry-After column), applies clear pass/fail rules, and saves a timestamped report under Reports/Session-07/ that previous runs never overwrite.

One engine, two scenarios — different verdicts. The shared mechanics (clean → fire one burst of distinct requests → record 2xx / 429 + Retry-After → count rows) live once in FloodCore. Each scenario reuses that engine but brings its own verdict. The only difference between BEFORE and AFTER is whether the rate limiter is switched on in the server — which is a reader step below, exactly like the Day-05 controller edit.

⚖️ Lesson 7 has TWO scenarios with different verdict logic

Both fire the same burst of distinct requests. What differs is what counts as success:

① BEFORE — Unprotected flood

dotnet run --project tools/Wassal.SessionTests -- session-07-unprotected-flood

A diagnostic. Its job is to show there's no defence, so it's happy when everything gets through. PASS when:

Nothing was shed (0 × 429)
Every request accepted (accepted == total)
DB rows == accepted (each made one order)

Meaning: the server has no backpressure. A bot could soak all capacity.

② AFTER — Rate limited

dotnet run --project tools/Wassal.SessionTests -- session-07-rate-limited

Short alias: session-07. A proof. Happy only when the server shed load and stayed correct and cooperated. PASS when:

Some 429s (load shed) & some 2xx (not a brick wall)
Every 429 carried a Retry-After
DB rows == accepted (rejected did no work)

⚠️ Run the AFTER scenario before adding the limiter and it will (correctly) fail — nothing was shed, so it looks exactly like BEFORE. That's the same rule as every session: match the scenario to the state of the server. Unprotected scenario on the undefended build; rate-limited scenario on the protected build.

Task 1 — Flood the Undefended Server (BEFORE) (10 min)

Start the API exactly as it is today — the idempotent server from Lessons 5 & 6, with no rate limiter:

dotnet run --project src/Wassal.Monolith --launch-profile https

Now fire a burst of 30 distinct requests at once and see how many the server accepts:

# From the repo root
dotnet run --project tools/Wassal.SessionTests -- session-07-unprotected-flood

Optional overrides (defaults come from appsettings.json):

dotnet run --project tools/Wassal.SessionTests -- session-07-unprotected-flood `
    --requests 30 `
    --cleanDb true

The console prints a per-request table (with a Retry-After column, all dashes for now) and a summary, then saves a report like Reports/Session-07/Session-07-Unprotected-Flood-RunReport-2026-06-21-143012.txt.

🐛 No backpressure. You'll see 30 accepted, 0 rejected, and 30 orders in the DB. The server shed nothing — it dutifully did the work for every request in the burst. Now imagine the burst is 30,000 from a bot: the server still says "yes" to all of them, until it falls over. Idempotency kept the data correct, but there is no defence against volume. Correct, and on fire.

💡 This scenario passes (exit 0) when the storm gets through — that's the point of a BEFORE diagnostic. "Nothing shed + DB rows == accepted" is the documented undefended state we want to make visible before we fix it.

✅ Done when: the flood shows 0 × 429 and DB rows == accepted == total (the server accepted everything).

Task 2 — Add a .NET 8 Token-Bucket Rate Limiter (15 min)

The fix lives entirely on the server, and .NET 8 ships the rate limiter in the framework — no Polly, no NuGet package. We register a single named token bucket and apply it to the orders endpoint.

First, register the limiter in Program.cs (add the usings, then the service):

// src/Wassal.Monolith/Program.cs (additions)
using System.Threading.RateLimiting;
using Microsoft.AspNetCore.RateLimiting;

// ... after builder.Services.AddControllersWithViews(); ...

builder.Services.AddRateLimiter(options =>
{
    // One named token bucket for the orders endpoint.
    options.AddTokenBucketLimiter("orders", o =>
    {
        o.TokenLimit          = 5;                        // bucket holds 5 tokens (max burst)
        o.TokensPerPeriod     = 5;                        // refill 5 tokens...
        o.ReplenishmentPeriod = TimeSpan.FromSeconds(1);  // ...every 1 second
        o.QueueLimit          = 0;                        // don't queue — reject immediately
        o.AutoReplenishment   = true;
    });

    // What the server sends when a request is rejected: 429 + Retry-After.
    options.RejectionStatusCode = StatusCodes.Status429TooManyRequests;
    options.OnRejected = async (context, token) =>
    {
        // The token bucket can estimate when a token will next be free.
        if (context.Lease.TryGetMetadata(MetadataName.RetryAfter, out var retryAfter))
            context.HttpContext.Response.Headers.RetryAfter =
                ((int)retryAfter.TotalSeconds).ToString();

        await context.HttpContext.Response.WriteAsync(
            "Too many requests. Please retry after a moment.", token);
    };
});

Then enable the middleware in the pipeline (after UseRouting, before the endpoints):

// src/Wassal.Monolith/Program.cs (pipeline)
app.UseRouting();

app.UseRateLimiter();   // ← add this line

app.UseAuthorization();

Finally, attach the named limiter to the order-creation action:

// src/Wassal.Monolith/Controllers/OrdersController.cs
using Microsoft.AspNetCore.RateLimiting;   // at the top

// ...

[EnableRateLimiting("orders")]   // ← apply the "orders" bucket to this endpoint
[HttpPost]
public async Task<IActionResult> Create([FromBody] CreateOrderRequest req)
{
    // ... unchanged idempotency logic from Lessons 4 & 5 ...
}

Two things worth noticing:

QueueLimit = 0 means we reject immediately instead of making callers wait in a queue — that's load shedding, the fail-fast behaviour from Mental Model 1.
We set RejectionStatusCode to 429 (the default is 503) and add Retry-After in OnRejected — turning a bare rejection into the cooperative contract.

Restart the app so the limiter is live:

dotnet run --project src/Wassal.Monolith --launch-profile https

💡 Why a token bucket and not "X per second" flat? A flat counter either forbids all bursts (bad for normal traffic that clumps) or resets on a hard boundary (letting a double-burst slip through at the edges). The token bucket allows a small, controlled burst (the bucket size) on top of a steady refill rate — the sweet spot for real APIs. We teach this one algorithm only; fixed/sliding windows exist but are out of scope today.

✅ Done when: the app rebuilds with no errors and the orders endpoint is decorated with [EnableRateLimiting("orders")].

Task 3 — Prove Load Shedding (AFTER) (10 min)

Run the exact same burst — only the server changed:

# SAME runner, SAME burst engine — different verdict
dotnet run --project tools/Wassal.SessionTests -- session-07-rate-limited

# Short alias:
dotnet run --project tools/Wassal.SessionTests -- session-07

This scenario applies 4 explicit pass/fail checks:

Load shed (some 429s) — the server rejected the excess
Some requests still succeeded — it's a limiter, not a brick wall
Every 429 carried Retry-After — the cooperative contract held
Rejected did no work (DB rows == accepted) — shed requests created nothing

✅ Sheds load AND stays correct. With a bucket of 5, a burst of 30 yields roughly 5 accepted (200) and ~25 rejected (429), and every one of those 429s carries a Retry-After value in the table. The DB holds exactly as many orders as were accepted — the rejected requests did no work, so there's no half-finished mess to clean up. The server protected itself, served whoever it could, and told the rest precisely when to return.

⚠️ Exact counts will wiggle. Because the bucket refills every second, a slightly slower burst may let a few extra through. The verdict doesn't pin an exact number — it checks the shape: some shed, some served, every 429 cooperative, and the DB matching the accepted count. That's robust across machines.

✅ Done when: all 4 checks PASS — some 429s with Retry-After, some 2xx, and DB rows == accepted.

Task 4 — Close the Loop: 429 + Retry-After Meets Backoff (10 min)

Put Lessons 6 and 7 side by side and the whole picture clicks:

SERVER (Lesson 7):  "I'm full. 429. Retry-After: 1."
CLIENT (Lesson 6):  reads Retry-After → waits ~1s → retries
                    with the SAME Idempotency-Key
SERVER:             bucket refilled → 200 OK, replayed=false → 1 order

Three layers, three jobs:
  Idempotency   → the retry is SAFE          (no duplicate order)   [L3–5]
  Backoff+jitter→ the client is GENTLE        (no client-side storm) [L6]
  Rate limiting → the server is PROTECTED     (no server overload)   [L7]

Notice how the pieces depend on each other. Rate limiting causes rejections, which require the client to retry, which is only safe because the server is idempotent, and only gentle because the client backs off — using the Retry-After the server handed it. None of the three lessons stands alone; together they are the real shape of a system that survives a Friday-night surge.

💡 🔜 Continued in Day 7 · Part 2 — The Polite Client Obeys Retry-After. Part 1 (here) taught the server to return 429 + Retry-After. Part 2 teaches the client to read Retry-After, wait at least that long, and retry the same operation with the same Idempotency-Key — turning the 25 rejected requests into eventual successes instead of lost work. Same Day 7, second half.

✅ Done when: you can explain how a single 429 + Retry-After connects all three layers (idempotency, backoff, rate limiting).

Task 5 — Capture the Evidence, Tag, Document (5 min)

The BEFORE and AFTER reports under Reports/Session-07/ are your evidence — attach both to the writeup. Then commit the server change + scenarios + session plan and tag the AFTER state:

cd D:\books\distributed-system\Wassal
git add src/Wassal.Monolith/ tools/Wassal.SessionTests/ `
        docs/sessions/Day-07-Rate-Limiting-Load-Shedding.html
git commit -m "feat: Lesson 7 — token-bucket rate limiting + load shedding (429 + Retry-After)"
git tag lesson-07-after

Suggested lesson writeup — docs/lessons/lesson-07-rate-limiting-load-shedding.md:

# Lesson 7 — Rate Limiting & Load Shedding

**Tags:** `lesson-07-before` (unprotected) → `lesson-07-after` (rate limited)
**Builds on:** Lesson 6 (`lesson-06-after`)

## The pain (BEFORE)

With no limiter, a burst is accepted in full — the server has no
backpressure. Idempotency keeps the data correct, but one rude client
can soak all capacity and starve everyone else. Correct, and on fire.

## The fix (AFTER)

A .NET 8 built-in token-bucket limiter (5 tokens, +5/sec, QueueLimit 0)
on the orders endpoint. Excess requests get 429 + Retry-After; the load
is shed fast; accepted requests still each create exactly one order
(rejected requests do no work). Verified by `session-07-rate-limited`.

| Scenario                  | Shed?      | Correctness         | Caller told when? |
|---------------------------|------------|---------------------|-------------------|
| Unprotected flood         | no (0×429) | 1 order/accepted ✅ | no                |
| Rate limited (token bucket)| yes (429) | 1 order/accepted ✅ | Retry-After ✅     |

## The contract

429 + Retry-After is the signal a Lesson-6 client backs off on. Server
sets the pace; client respects it. Idempotency makes the eventual retry
safe.

## What's still deliberately missing (later lessons)

- **The client side of the handshake** — a client that honours Retry-After,
  retries with the same key, and ends with one order. Now covered in
  **Day 7 Part 2 — The Polite Client Obeys Retry-After**.
- **Distributed rate limiting** across replicas (shared counter / Redis) —
  belongs with the scale-out lessons. Today's limiter is per-instance.
- **Per-user / per-API-key limits, quotas, auth** — one global bucket for now.
- **Polly, circuit breaker** — Part 6.
- **Key TTL / expiry** — still parked for the caching lesson.

✅ Done when: both Session-07 reports are saved, the limiter + scenarios + session plan are committed, and lesson-07-after is tagged.

Quick Checklist

#	Task	Time	Done When
1	Flood the undefended server → expose no backpressure	10 min	0 × 429, DB rows == accepted == total
2	Add the .NET 8 token-bucket limiter (429 + Retry-After)	15 min	App rebuilds, endpoint has `[EnableRateLimiting]`
3	Re-run the flood → witness load shedding	10 min	All 4 checks PASS (shed + correct + cooperative)
4	Connect 429 + Retry-After to Lesson 6 backoff	10 min	You can explain all three layers together
5	Save reports, commit, tag `lesson-07-after`	5 min	Both reports saved + writeup committed

🎓 The compounding effect — 7 lessons in. Your order endpoint now survives restarts and body tampering (L4), concurrency (L5), impolite-but-honest retries (L6), and now floods and bots (L7). Idempotency keeps it correct; backoff keeps the client gentle; rate limiting keeps the server standing. That trio — safe retries + client backoff + server backpressure — is exactly how production APIs survive a surge. Same endpoint, seven layers of deliberate, testable defence.

➡️ Next — Day 7 · Part 2: The Polite Client Obeys Retry-After

The server now says "not now" with 429 + Retry-After. Part 2 teaches the client to listen: read Retry-After, wait, and retry the same operation with the same Idempotency-Key until it completes.

Part of the **Fundamentals of Distributed Systems* series — building Wassal, a distributed food-delivery lab, one concept at a time.*

Distributed Systems in .NET — Day 6

mohamed Tayel — Sat, 25 Jul 2026 20:25:49 +0000

Session 5 made the server safe for retries. Today we move to the client side and learn to retry politely.

A retry is a real request — and a client that retries badly can bury a server that was about to recover. In this session we simulate several clients retrying one order the naive way (no waiting) and watch the attempts pile into one tiny window (a retry storm). Then we give them a bounded retry loop with exponential backoff and jitter, and watch the same number of attempts spread calmly over time.

📘 Where we left off in Lesson 5: You made idempotency correct under genuine concurrency. 10 same-key POSTs arriving at the same instant now produce 10 OK responses, 1 unique Order Id, 9 replayed=true, and exactly 1 DB row — no more HTTP 500s for the race losers. Tagged as lesson-05-after. But every "retry" so far was faked by us firing parallel requests. We never built the thing that actually retries in real life — the client. The server is now a safe place to retry into. Today we ask: how should the client retry, so that being safe doesn't accidentally become being slow — or being an outage?

🎯 The one idea to take away: Idempotency makes retries safe for correctness. Backoff & jitter make retries safe for system load. Correctness is not the same as scalability. You need both, and they are solved in different places: one on the server (Lesson 5), one on the client (today).

What you'll reinforce and add

🔁 Reinforce — from Lessons 4 & 5:

Idempotency-Key = one key per business operation
replayed=true means "I returned the existing result"
DB row count as the authoritative dedup check
Before/After discipline + the .NET scenario runner
Reading a console table + saved report

✨ New — added today:

Seeing a retry as another real HTTP request
Timeout vs retry vs backoff vs jitter
The thundering-herd / retry-storm failure mode
A hand-written exponential-backoff + jitter loop
Why the SAME key must survive across retries

🎯 Today's goal: Simulate several clients that each retry one order. First let them retry the naive way — no waiting — and watch all the retries pile into one tiny window (a retry storm). Then give them a bounded retry loop with exponential backoff and jitter, reusing the same Idempotency-Key per operation, and watch the same number of attempts spread calmly over time. In both runs the database stays correct (one order per client) — proving the point: the fix here is not about correctness, it's about load.

🧠 Mental Model 1 — A retry is another real HTTP request

The most common beginner mistake is to imagine a "retry" as some lightweight, free thing the framework does in the background. It is not. When a client retries, it opens a real connection and sends a real POST. The server does real work to answer it.

One logical operation: "Create my Koshary + Pizza order"

Attempt 1  ──►  POST /api/orders   (real request, real CPU, real DB hit)
   timeout / no answer in time
Attempt 2  ──►  POST /api/orders   (ANOTHER real request)
   still slow...
Attempt 3  ──►  POST /api/orders   (ANOTHER real request)

3 attempts  =  3 real requests the server must handle.
1 order      =  what the user actually wanted.

So if 1,000 clients each retry 5 times, that is not 1,000 requests — it is up to 5,000. Retries multiply load. Idempotency makes sure those 5,000 requests still create only 1,000 orders (correctness). It does nothing to reduce the 5,000 (load). That gap is exactly what today is about.

💡 Why does a client even retry? Usually because of a timeout: the request was sent, but no answer came back in time. Here is the cruel part — the client cannot tell the difference between "the server never got it" and "the server did the work but the reply got lost." So a careful client must retry to be safe… which is only OK because the server is idempotent. The two halves depend on each other.

🧠 Mental Model 2 — Timeout vs Retry vs Backoff vs Jitter

Four words that beginners often blur together. They are four different decisions:

TIMEOUT  → "How long do I wait for one answer before I give up on it?"
            e.g. wait 2s; if no response, treat this attempt as failed.

RETRY    → "After a failed attempt, do I try again, and how many times?"
            e.g. try at most 5 times, then surface an error to the user.

BACKOFF  → "How long do I WAIT between retries?"
            naive  : 0ms, 0ms, 0ms ...      (hammer immediately)
            backoff: 250ms, 500ms, 1s, 2s   (wait longer each time)

JITTER   → "Do all clients wait the EXACT same amount?"
            no jitter: every client retries at t=250ms  → they sync up
            jitter   : each waits 250ms ± a random wobble → they spread out

Put simply with a teaching analogy: imagine a door that jams. Timeout is how long you push before letting go. Retry is deciding to push again. Backoff is waiting a bit longer before each push instead of slamming it nonstop. Jitter is making sure you and everyone else in the crowd don't all shove at the exact same second — because that is what breaks the door.

⚠️ Backoff without jitter is only half a fix. If every client backs off by exactly 250ms, then 500ms, then 1s, they stay perfectly synchronized — the herd just stampedes in waves. Jitter is the small random offset that breaks the lockstep so the load smears out instead of arriving in spikes.

🧪 Verification Discipline — same rule, new session

As always, we do not prove this with throwaway PowerShell. Session 6 gets its own repeatable scenarios under tools/Wassal.SessionTests, sharing the same engine and reporting as Session 5. Each scenario cleans the DB, fires the requests in a repeatable way, prints a readable console table, applies clear pass/fail rules, and saves a timestamped report under Reports/Session-06/ that previous runs never overwrite — so the evidence is easy to attach to this writeup.

One engine, two scenarios — different verdicts. The shared mechanics (clean → simulate N clients × A attempts reusing one key per client → record when each attempt was sent → count rows) live once in RetryStormCore. Each scenario reuses that engine but brings its own verdict. The only behavioural difference between the two is the retry timing policy — naive (no wait) vs resilient (backoff + jitter). No server change is needed between them.

⚖️ Lesson 6 has TWO scenarios with different verdict logic

Both simulate the same clients making the same number of attempts with the same reused key. What differs is what counts as success:

① BEFORE — Naive retry storm

dotnet run --project tools/Wassal.SessionTests -- session-06-naive-retry-storm

A diagnostic. Its job is to expose concentrated load, so it is happy when the attempts pile up. PASS when:

Attempts are crammed together (avg spacing < 50 ms)
Many attempts land in one short window (load concentration)
Duplicates still avoided (DB rows = number of clients)

Meaning: idempotency kept it correct, but the retries arrived as one spike. Correctness ≠ scalability.

② AFTER — Resilient retry

dotnet run --project tools/Wassal.SessionTests -- session-06-resilient-retry

Short alias: session-06. A proof. It is happy only when correctness and spread both hold. PASS when:

DB row count == clients, unique ids == clients
Every client eventually succeeded
Retries spread out (avg spacing ≥ 50 ms)

⚠️ Same input, opposite verdicts — on purpose. The 50 ms boundary is the hinge: the naive run sits below it (a spike), the resilient run sits above it (spread out). Both keep the database correct — that is the whole lesson. The improvement you are proving is about load, not data.

Task 1 — Feel the Naive Retry Storm (BEFORE) (10 min)

Make sure the Wassal API is running (the same idempotent server from Lesson 5 — no changes needed):

dotnet run --project src/Wassal.Monolith --launch-profile https

Now run the naive scenario. It simulates 5 clients, each retrying its order 5 times, with no wait between attempts:

# From the repo root
dotnet run --project tools/Wassal.SessionTests -- session-06-naive-retry-storm

Optional overrides (defaults come from appsettings.json):

dotnet run --project tools/Wassal.SessionTests -- session-06-naive-retry-storm `
    --clients 5 `
    --attempts 5 `
    --cleanDb true

The console prints a per-client summary, an attempt timeline sorted by send time (so you can literally see them stack up), and a metrics block. It then saves a report like Reports/Session-06/Session-06-Naive-Retry-Storm-RunReport-2026-06-21-143012.txt.

🐛 The storm. You'll see all 25 attempts land almost on top of each other — a tiny avg spacing (single-digit ms) and a high Max attempts in 200ms window. The database is still correct (5 orders, one per client, duplicates avoided) because each client reused its key. But the load was concentrated into one spike. On a server that is busy or just recovering from a blip, that spike is the thundering herd that turns a hiccup into a full outage.

💡 Read the verdict carefully. This scenario passes (exit 0) when the storm IS observed — that's the point of a BEFORE diagnostic. "Attempts too close together + duplicates still avoided" is the documented bad state we want to make visible before we fix it.

✅ Done when: you've seen the attempts cram into one window, a tiny avg spacing, and DB rows = number of clients (duplicates avoided).

Task 2 — Write a Hand-Made Backoff + Jitter Loop (15 min)

The fix is entirely on the client. We change one thing: how long a client waits before each retry. We keep it hand-written on purpose — no Polly, no circuit breaker — so the mechanics stay visible. Here is the whole idea in a few lines:

// the resilient retry policy (illustrative)

// One key for the whole operation — generated ONCE, reused on every retry.
var idempotencyKey = Guid.NewGuid().ToString();

const int baseDelayMs = 250;     // first backoff gap
const double factor   = 2.0;     // double the wait each time (exponential)
const int maxAttempts = 5;       // bounded — never retry forever

var rng = new Random();

for (var attempt = 1; attempt <= maxAttempts; attempt++)
{
    // 1) WAIT before retrying (the first attempt waits 0).
    if (attempt > 1)
    {
        var gap    = baseDelayMs * Math.Pow(factor, attempt - 2); // 250, 500, 1000, 2000
        var jitter = 0.6 + rng.NextDouble() * 0.8;                // random ×[0.6 .. 1.4)
        await Task.Delay((int)(gap * jitter));                    // backoff + jitter
    }

    // 2) Send the SAME request with the SAME key.
    var result = await PostOrderAsync(body, idempotencyKey);

    // 3) Stop as soon as it works; otherwise loop and back off again.
    if (result.Ok) return result;
}

// Bounded: after maxAttempts we give up gracefully (surface a clean error,
// not an infinite stampede).

Three things make this "resilient" instead of "naive":

Bounded — maxAttempts means we never retry forever.
Exponential backoff — each gap is roughly double the last (250ms → 500ms → 1s → 2s), so a struggling server gets breathing room that grows.
Jitter — the ±40% random multiplier means clients don't retry in lockstep, so the load smears out instead of arriving in synchronized waves.

💡 In this repo the exact same policy already lives in the scenario class Session06ResilientRetryScenario (method ResilientDelay), and the naive one in Session06NaiveRetryStormScenario (NaiveDelay returns 0). The shared firing engine is RetryStormCore. You don't have to edit the server at all — the difference between BEFORE and AFTER is purely this delay function.

✅ Done when: you can explain, in one sentence each, what bounded / backoff / jitter add — and why the key is created once, not per attempt.

Task 3 — Prove Resilient Retry (AFTER) (10 min)

Run the resilient scenario — same clients, same attempts, same per-client key, only the timing policy changed:

# SAME runner, SAME engine — different policy, different verdict
dotnet run --project tools/Wassal.SessionTests -- session-06-resilient-retry

# Short alias:
dotnet run --project tools/Wassal.SessionTests -- session-06

This scenario applies 5 explicit pass/fail checks:

DB row count == clients — one order per client
Unique ids == clients — every client's retries point at its one order
Duplicates avoided — idempotency held under retries
All clients succeeded — every operation eventually got a result
Retries spread out — average spacing ≥ 50 ms (backoff actually applied)

✅ Both halves hold. Read it against the timeline: the same 25 attempts are now smeared across a few seconds instead of one burst. avg spacing jumps from single-digit ms to well over the 50 ms boundary, and Max attempts in 200ms window drops sharply. Meanwhile the DB still shows one order per client. Idempotency kept it correct; backoff + jitter kept it gentle. Exit code 0 means all five checks passed.

⚠️ If you ran the naive scenario after this, it would "fail" its storm check — because there's no storm left to find. That failure is good news. Always match the scenario to what you're testing: naive to expose the spike, resilient to prove the spread.

✅ Done when: all 5 checks PASS — duplicates avoided AND retries spread out (avg spacing ≥ 50 ms).

Task 4 — Why the Same Idempotency-Key Must Survive Every Retry (10 min)

This is the single most important rule of client retries, and the easiest to get wrong. The Idempotency-Key belongs to the operation, not to the attempt. The client must generate it once, before the first try, and reuse the exact same value on every retry of that operation.

✅ CORRECT — key created once, reused on every retry

operation: "create my order"   →  key = ABC-123   (generated ONE time)
   attempt 1  →  POST  Idempotency-Key: ABC-123
   attempt 2  →  POST  Idempotency-Key: ABC-123     ← same key
   attempt 3  →  POST  Idempotency-Key: ABC-123     ← same key
Server: "I've seen ABC-123 → replay the existing order."  → 1 order ✅

Because the key is stable, the server recognizes attempts 2 and 3 as the same intent and replays the first result. That is exactly the replayed=true behaviour from Lessons 4 and 5 — and it's why our resilient scenario produces one order per client no matter how many times each client retries.

⚠️ The anti-pattern: a new key per retry. If the client generates a fresh key for each attempt, every retry looks like a brand-new operation to the server — and idempotency can't help, because dedup is keyed on… the key.

❌ WRONG — a new key every attempt

operation: "create my order"
   attempt 1  →  POST  Idempotency-Key: KEY-1   →  Server: new! → Order #55
   attempt 2  →  POST  Idempotency-Key: KEY-2   →  Server: new! → Order #56
   attempt 3  →  POST  Idempotency-Key: KEY-3   →  Server: new! → Order #57
Result: 1 user intent → 3 DUPLICATE orders (and 3 charges). 💥

Here the customer wanted one Koshary + Pizza order and got billed three times. The server did nothing wrong — it was told, three times, "this is a new order." Generating a new key per retry doesn't just weaken idempotency; it completely defeats it. All the work from Lessons 3–5 is undone by one client-side mistake.

💡 Rule of thumb. New key = new intent. Same retry = same key. If you can't point to where the key was generated once and held across retries, your "idempotent" client isn't. In our runner, that's exactly why each simulated client builds one key (…-client-N) and reuses it for all of its attempts.

✅ Done when: you can state why reusing the key gives 1 order, and why a new key per retry gives N duplicates.

Task 5 — Capture the Evidence, Tag, Document (5 min)

Unlike Lesson 5, there is no server change today — the BEFORE and AFTER are two client policies run against the same idempotent API. So the "before/after" lives in the two saved reports under Reports/Session-06/, not in a controller diff. Attach both reports to the writeup as evidence.

Commit the new scenarios + session plan, and tag the AFTER state:

cd D:\books\distributed-system\Wassal
git add tools/Wassal.SessionTests/ docs/sessions/Day-06-Timeouts-Retries-Backoff-Jitter.html
git commit -m "feat: Lesson 6 — client-side resilient retries (backoff + jitter) + scenarios"
git tag lesson-06-after

Suggested lesson writeup — docs/lessons/lesson-06-timeouts-retries-backoff-jitter.md:

# Lesson 6 — Timeouts, Retries, Backoff & Jitter

**Tags:** `lesson-06-before` (naive) → `lesson-06-after` (resilient)
**Builds on:** Lesson 5 (`lesson-05-after`)

## The pain (BEFORE)

Naive retries (no wait) from several clients pile every attempt into one
tiny window. Idempotency still keeps the DB correct — one order per client,
duplicates avoided — but the LOAD is concentrated into a spike: the
thundering herd. Correct is not the same as scalable.

## The fix (AFTER)

A bounded, hand-written retry loop with exponential backoff (250ms × 2 each
time) and ±40% jitter, reusing ONE Idempotency-Key per operation. Same
attempts, same correctness, but the load is spread over time instead of
stacked. Verified by `session-06-resilient-retry`: DB rows = clients,
unique ids = clients, every client succeeded, avg spacing ≥ 50ms.

| Scenario                       | Correctness        | Load                         |
|--------------------------------|--------------------|------------------------------|
| Naive retry (no wait)          | 1 order/client ✅  | concentrated spike ❌        |
| Resilient (backoff + jitter)   | 1 order/client ✅  | spread over time ✅          |

## The key rule

Generate the Idempotency-Key ONCE per operation and reuse it on every
retry. A new key per retry turns 1 intent into N duplicate orders and
defeats everything from Lessons 3–5.

## What's still deliberately missing (later lessons)

- **Polly / resilience libraries.** We hand-rolled the loop to see it.
  Polly comes later (Part 6, Circuit Breakers).
- **Circuit breaker.** Stop calling a service that is clearly down. Later.
- **Server-side rate limiting / 429.** Defending the server FROM abusive
  clients is a separate lesson (Part 6).
- **Key TTL / expiry.** Still parked for the caching lesson.

✅ Done when: both Session-06 reports are saved, the scenarios + session plan are committed, and lesson-06-after is tagged.

Quick Checklist

#	Task	Time	Done When
1	Run naive storm → expose concentrated load	10 min	Tiny avg spacing + DB rows = clients (duplicates avoided)
2	Understand + write the backoff + jitter loop	15 min	You can explain bounded / backoff / jitter
3	Run resilient retry → prove the fix	10 min	All 5 checks PASS (correct AND spread out)
4	Why reuse the key; why a new key per retry breaks it	10 min	Can explain 1 order vs N duplicates
5	Save reports, commit, tag `lesson-06-after`	5 min	Both reports saved + writeup committed

🎓 The compounding effect — 6 lessons in. Your idempotent POST endpoint now survives app restarts (Lesson 4), body tampering (Lesson 4), and genuine concurrency (Lesson 5) — and you finally have the client that knows how to retry into it safely (Lesson 6). The server guarantees correctness; the client guarantees politeness. That two-sided contract — at-least-once delivery + idempotent handling + backoff — is the real shape of "exactly-once effect" that production systems actually ship.

Part of the **Fundamentals of Distributed Systems* series — building Wassal, a distributed food-delivery lab, one concept at a time.*