Day 15 decided who owns an outbox message. But leasing says nothing about what happens when processing keeps failing. A message that can never succeed — a poison message — will be retried forever unless retry has a limit, a backoff, and an exit.
🙏 Credit where it's due — This series is my attempt to internalize and share what I learned from Mahmoud Youssef's excellent course, Fundamentals of Distributed Systems on Udemy. The course material, structure, and topic flow are his work; the explanations, code examples, and diagrams in these articles are my own rewrite in my own words. If you find this useful, please consider taking the course — it goes much deeper than these articles can.
Where we are — after Days 13, 14 and 15
- Day 13 — Durable Reconciliation: the pending work survives a restart.
- Day 14 — Transactional Outbox: the payment row and the outbox row commit together, atomically.
- Day 15 — Outbox Leasing: exactly one worker owns a message at a time, and ownership expires if the worker dies.
- Day 16 (this document): leasing decides who processes a message — not what happens when processing keeps failing. A poison message needs bounded retries, backoff, and a dead-letter exit.
🎯 The one idea to take away: Leasing says who owns the work now. Retry says when to try again. Dead-letter says when to stop and ask a human. A durable, atomic, leased message can still fail every time it is processed. Without a limit and an exit, one bad message is retried forever and can starve the whole queue.
1. Why a leased, durable outbox can still loop forever
Everything so far assumed processing eventually succeeds. Real provider calls do not. They fail two ways:
- Transient failure — a timeout, a blip, a temporary 503. Another try later will probably work.
Permanent failure — a malformed record, a rejected account, a bug. This message will never succeed. It is a poison message.
✅ Day 15 solved — ownership: one worker owns a message at a time; a dead owner's lease expires and another worker takes over.
❌ Day 16 solves — when to stop: a naive "retry until it works" loop never stops on a poison message — it re-claims and re-fails forever.
⚠️ The lease actually makes it worse. Because a message returns to the queue after each failure (or after its lease expires), the poison message is guaranteed to come back — forever — unless something counts the attempts and calls a halt.
2. The BEFORE scenario — retried forever, and the state hides no lie but never ends
session-16-poison-message-retried-forever seeds the Day 14 durable outbox (6 payments + 6 pending messages), then a single worker processes them with the most naive policy: on failure, put the message straight back to Pending and try again — no limit, no backoff, no exit. One message, PAY-003, is a poison message: its provider status check fails every time.
-
healthy ×5:
PAY-001, 002, 004, 005, 006each processed once →Processed(4 Paid, 1 Failed). -
PAY-003 ×∞: claim → status check FAILS → back to
Pending→ claim → FAILS → … the attempt count climbs with no ceiling. -
demo cap: a real run never ends, so the scenario stops after 10 retries and inspects the state: still
Pending,AttemptCount = 10, neverProcessed, no dead-letter to catch it.
From the real run: healthy processed 5/5, poison retries 10, poison AttemptCount 10, poison final status Pending, provider status-check calls 15 (of which 10 failed), dead-lettered 0.
⚠️ Verdict from the real report: "A POISON MESSAGE RETRIED FOREVER — NO LIMIT, NO EXIT ❌". Nothing here is a lie or a double-charge — it is unbounded, wasted work that a shared queue would let starve every other message.
3. The retry columns and the new terminal state
The AFTER scenario adds a retry policy to the leased outbox row and one new status. A message now ends in one of two terminal states — Processed or DeadLettered:
Pending ─claim─▶ Processing ─success─▶ Processed (terminal, good)
│
├─fail, attempts < max ─▶ Pending + NextAttemptUtc (backoff, retry later)
└─fail, attempts ≥ max ─▶ DeadLettered + LastError (terminal, parked)
| Column | Meaning |
|---|---|
AttemptCount / MaxAttempts
|
How many times we've tried, and the ceiling before giving up. |
NextAttemptUtc |
Backoff gate — the message is not claimable again until this time. |
LastError |
Why the last attempt failed — recorded for the human who inspects a dead letter. |
Status is now Pending | Processing | Processed | DeadLettered. The DeadLettered state is terminal but not lost — the message is parked, out of the worker's way, with its error attached.
4. The claim, now gated by backoff
The Day 15 atomic claim gets one extra clause: a message is only claimable if its backoff has elapsed. The claim stays a single BEGIN IMMEDIATE transaction with a conditional update:
BEGIN IMMEDIATE;
SELECT Id FROM OutboxMessages
WHERE Status IN ('Pending','Processing')
AND (Status='Pending' OR LockedUntilUtc < $now) -- lease (Day 15)
AND (NextAttemptUtc IS NULL OR NextAttemptUtc <= $now) -- backoff (Day 16)
ORDER BY CreatedAtUtc LIMIT 1;
UPDATE OutboxMessages SET Status='Processing', LockedBy=$worker,
LockedUntilUtc=$lease, AttemptCount = AttemptCount + 1
WHERE Id=$id AND … (same conditions) …;
COMMIT;
On failure the worker chooses one of two paths, purely from AttemptCount vs MaxAttempts:
if (AttemptCount < MaxAttempts)
Status='Pending', NextAttemptUtc = now + backoff(AttemptCount), LastError=… // retry later
else
Status='DeadLettered', LastError=… // give up, park it
5. Backoff — and proving it without waiting
Backoff spreads retries over time instead of hammering a struggling provider. Here the schedule is exponential: 200 ms × 2^(n-1) → 200 ms, then 400 ms. A message in backoff is genuinely not claimable — the worker's claim returns nothing, so it advances a deterministic ScenarioClock to the next attempt time (no real waiting). The real run's timeline for PAY-003:
- t = 09:00:00.000Z: attempt 1/3 FAILED → backoff 200 ms → NextAttempt = 09:00:00.200Z (not claimable until then).
- t = 09:00:00.201Z: attempt 2/3 FAILED → backoff 400 ms → NextAttempt = 09:00:00.601Z.
- t = 09:00:00.602Z: attempt 3/3 FAILED → limit reached → DEAD-LETTERED (LastError recorded).
💡 Deterministic by design. Because the clock is injected, the backoff windows are exact and the test never sleeps. In production the same gate is a real wall-clock comparison; the logic is identical.
6. Dead-letter — parked, not lost
After MaxAttempts, the poison message moves to DeadLettered — a terminal state that takes it out of the worker's claim query so the queue keeps moving. Crucially, it is not deleted: its LastError is kept for a human or a redrive tool.
- 🔁 A retry loop says — never stop: keep trying the same failing work forever — burning provider calls and blocking the queue.
- ☠️ A dead-letter says — stop and surface: after N tries, park the message with its error so people can see it — and let everything else proceed.
💡 The payment is honestly still pending. Dead-lettering the message does not fake a payment outcome —
PAY-003's payment row staysPendingConfirmation. We stopped the automatic retry; we did not invent a result.
7. The real implementation — proven by running it
Day 16 reuses the Day 14 outbox (OutboxSqliteCore) and the Day 15 lease model (Session15LeasingCore) unchanged, and adds a small retry core plus two scenarios registered in Program.cs:
-
BEFORE:
Session16PoisonRetriedForeverScenario→ commandsession-16-poison-message-retried-forever -
AFTER:
Session16RetryWithDeadLetterScenario→ commandsession-16-retry-with-dead-letter
No new packages — the same Microsoft.Data.Sqlite. Standard .NET only: the atomic claim via BEGIN IMMEDIATE, a ScenarioClock for deterministic backoff windows, and a provider whose status check throws for the poison key (PAY-003). The poison message is a would-be Failed payment, so dead-lettering it loses no money.
Real BEFORE vs AFTER metrics
| Metric (real run) | BEFORE (naive) | AFTER (policy) |
|---|---|---|
| Healthy messages Processed | 5 / 5 | 5 / 5 |
| Poison attempts | 10 (capped; unbounded) | 3 (= MaxAttempts) |
| Retries with backoff | 0 (none) | 2 (200 ms, 400 ms) |
| Poison final outbox status | Pending (never ends) | DeadLettered |
| Dead-lettered messages | 0 (no such concept) | 1 (LastError recorded) |
| Provider status-check calls | 15 (10 failed) | 8 (3 failed) |
| Outbox Pending / Processing at end | 1 / 0 (stuck) | 0 / 0 (drained) |
| Resolved Paid / Failed (healthy) | 4 / 1 | 4 / 1 |
| Duplicate payment / outbox rows | 0 / 0 | 0 / 0 |
| Fake successes | 0 | 0 |
The headline: the poison message goes from an unbounded loop (10 and climbing, 10 wasted provider calls) to exactly 3 attempts and a clean exit — and the queue ends fully drained instead of stuck with one Pending row forever.
Safety checks from the real AFTER report — all passing
poison tried exactly MaxAttempts → 3 / 3poison dead-lettered with LastError → status DeadLettered, error setqueue drained → 0 pending / 0 processingno duplicate payment/outbox rows → max 1 per keytx ids only from provider, Paid only → 4 Paid w/ id, 0 Failed w/ id
💡 Verdicts from the real reports. BEFORE: "A POISON MESSAGE RETRIED FOREVER — NO LIMIT, NO EXIT ❌". AFTER: "BOUNDED RETRY + DEAD-LETTER — THE QUEUE KEEPS MOVING ✅". Both exit with code
0only when every check passes.
Run it yourself
dotnet run --project tools/Wassal.SessionTests -- session-16-poison-message-retried-forever
dotnet run --project tools/Wassal.SessionTests -- session-16-retry-with-dead-letter
Each run prints the schema, the processing/backoff timeline, the per-message table, the ledgers, the checks, and the verdict, then saves a timestamped report under:
Reports/Session-16/Session-16-Poison-Message-Retried-Forever-RunReport-<timestamp>.txt
Reports/Session-16/Session-16-Retry-With-Dead-Letter-RunReport-<timestamp>.txt
The SQLite file is created in a per-scenario temp folder — e.g. %LOCALAPPDATA%\Temp\wassal-day16-deadletter-<random>\wassal-outbox.db — and cleaned up at the end; its path is printed in the report.
8. Final takeaway
- A durable, atomic, leased message can still fail every time — leasing decides ownership, not when to stop.
- Retry needs a limit (
MaxAttempts), a backoff (NextAttemptUtcgating re-claim), and an exit (DeadLettered). - A dead-lettered message is parked, not lost: terminal, out of the worker's way, with its
LastErrorfor a human. - Dead-lettering the message does not fake the payment — the payment stays honestly
PendingConfirmation. - All Day 14/15 guarantees still hold: one payment per key, one outbox row per payment, exclusive ownership, no duplicate charges, provider-only transaction ids.
The four days, side by side:
Day 13: The work survives a restart.
Day 14: The business state and outbox row commit together.
Day 15: Only one worker owns a message at a time.
Day 16: A message that always fails is tried a bounded number of times, then parked.
💡 Leasing says who owns the work now. Retry says when to try again. Dead-letter says when to stop and ask a human. Retry scheduling refinements, a hosted worker, and dead-letter redrive tooling are later lessons.
Part of the **Fundamentals of Distributed Systems* series — building Wassal, a distributed food-delivery lab, one concept at a time.*
Top comments (0)