Day 12 built a worker that resolves pending payments by asking the provider — but the pending work lived only in memory. Day 13 asks the harder question: what happens when the process crashes before that work runs? In-memory work disappears. Durable work survives.
🙏 Credit where it's due — This series is my attempt to internalize and share what I learned from Mahmoud Youssef's excellent course, Fundamentals of Distributed Systems on Udemy. The course material, structure, and topic flow are his work; the explanations, code examples, and diagrams in these articles are my own rewrite in my own words. If you find this useful, please consider taking the course — it goes much deeper than these articles can.
🧭 Where we are — after Days 11 and 12
-
Day 11 — Payment Fallback Without Lying: when the provider times out, we return an honest degraded response —
PendingConfirmation,PaymentConfirmed = false,RequiresStatusCheck = true, idempotency key preserved, no fabricated transaction id. "We do not know yet." -
Day 12 — Payment Reconciliation: a worker scans the pending store and calls
CheckPaymentStatus(originalKey)— a status read, never a re-charge — and moves each payment to the provider's real final state. "We checked the original attempt and learned the result." - Day 13 (this document): in Day 12 the pending store and the reconciliation queue lived in memory, inside one process. If that process crashes before the worker runs, the work is gone. Even if our application crashes, it must remember that the check is still required.
🎯 The one idea to take away: Pending is a promise. Durability makes sure the system remembers the promise. In-memory work disappears when the process dies. Durable work survives restarts. Knowing what to do is not enough — the work must outlive the process that created it.
1. Why in-memory background work is unsafe
Day 12's worker was correct — but it was only as durable as the process it ran in. A reconciliation queue held in a List<WorkItem> and a pending store held in a Dictionary share one fate: the moment the process exits, they are gone. Real processes exit all the time — a deploy, an OOM kill, a crash, a pod eviction, a machine reboot.
The dangerous window is small but real:
- A payment times out and becomes
PendingConfirmation,RequiresStatusCheck = true. - A reconciliation work item is enqueued in memory.
- The application crashes before the worker runs.
- The in-memory pending work disappears.
- After restart, the application does not remember what it must check.
✅ The worker (Day 12) — solved: knows exactly how to resolve a pending payment: CheckPaymentStatus(key), never re-charge.
❌ Remembering the work (Day 13 · still missing): the queue lives in memory. A restart erases it, and the recovered provider's answers are never collected.
💡 The failure mode here is not a double charge — it is forgetting. The provider may recover holding the real outcome, but our side no longer has a queued item asking about it. The follow-up obligation vanished with the process that created it.
2. Our application state vs the external provider's state
This distinction is the heart of the lesson. Two systems, two separate fates when our process crashes:
🧠 Our application state — volatile: pending store + reconciliation queue. Held in memory → lost on crash unless it is made durable.
🏦 The external provider — survives: a separate system. It never lost its records — it can be hung, then recover, still holding every original outcome.
Our application restarted.
The external payment provider did NOT lose its records.
In both scenarios the provider is modelled as its own object (ExternalPaymentProvider) created once and handed to both application instances — because a real provider does not restart when our process does. The gap Day 13 closes is entirely on our side: remembering that a check is still owed.
3. The BEFORE problem — the crash erases the work
The BEFORE scenario (session-13-in-memory-work-lost) replays Day 11/12, then models a real process boundary:
-
Phase 1 — Application Instance 1. 6 payments (
PAY-001…PAY-006) hit a hung gateway (2000 ms) behind theSemaphoreSlim(2)bulkhead with the 300 ms timeout. All 6 time out → 6PendingConfirmationrecords and 6 reconciliation work items, all in memory only. -
Simulated crash — Instance 1 is discarded. Every reference to its pending dictionary and its reconciliation queue is dropped and a brand-new Instance 2 is constructed — no shared references, no static state. This is what a crash does to memory, not a
list.Clear(). - Phase 3 — the provider recovers, still holding every answer (4 charged, 2 failed). But Instance 2's queue is empty. Work items after restart: 0. Status checks: 0. Resolved: 0. The obligation is gone.
From the real run: pending before crash 6, in-memory work items before crash 6, work items after restart 0, status checks after restart 0, resolved payments 0, still unresolved externally 6, duplicate charge attempts 0.
💡 Verdict from the real report: "THE WORKER KNEW WHAT TO DO — BUT THE CRASH ERASED THE WORK ❌". Every check in the BEFORE report passes — because the problem it proves is real: a correct worker with no durable memory forgets its obligations the instant the process dies.
4. The AFTER fix — a durable reconciliation store
The AFTER scenario (session-13-durable-reconciliation) changes one thing: where the pending work lives. When a payment becomes PendingConfirmation, its payment record and its reconciliation work item are persisted together as one JSON snapshot, written with a safe pattern:
write temporary file
flush / close (fsync — force the bytes to disk)
atomically move the temp file over the final file
- Instance 1 creates the payment attempts and persists each pending payment + work item durably.
- Simulated crash: all of Instance 1's in-memory state is discarded.
- Instance 2 starts with empty memory, reloads the six pending work items only from durable storage, and reconciles using status checks only.
From the real run, every payment reloaded and resolved to the provider's truth:
PAY-001 | PendingConfirmation → Paid | TX-PAY-001
PAY-002 | PendingConfirmation → Paid | TX-PAY-002
PAY-003 | PendingConfirmation → Failed | TransactionId=null
PAY-004 | PendingConfirmation → Paid | TX-PAY-004
PAY-005 | PendingConfirmation → Failed | TransactionId=null
PAY-006 | PendingConfirmation → Paid | TX-PAY-006
💡 This is outbox-shaped, not a production Transactional Outbox. A single JSON file plus an atomic replace buys us restart-survival — but it is not a database transaction that commits the business write and the outbox row together. A real transactional outbox, a hosted background worker, schedules, retries, and dead-letter handling are later lessons. Day 13 introduces the direction, honestly labelled.
5. The durable work-item model
One durable snapshot holds two lists — payments and their work items — persisted together so a restart never sees a payment without its follow-up obligation (or vice-versa):
| Durable payment record | Durable reconciliation work item |
|---|---|
IdempotencyKey |
WorkItemId |
PaymentStatus |
PaymentIdempotencyKey |
PaymentConfirmed |
Status — Pending | Completed |
RequiresStatusCheck |
AttemptCount |
TransactionId |
CreatedAt |
IsFallback |
CompletedAt (when completed) |
💡 Persisted together, as one logical snapshot. The payment state and its work item are written in a single atomic replace — so the durable store never records a pending payment whose reconciliation obligation was lost, which is exactly the failure the BEFORE scenario proved.
6. State transitions — before and after the restart
Every transition is driven by the provider's reported truth — reloaded from disk, never guessed:
Before crash: PendingConfirmation | WorkItem = Pending (durably saved)
After restart: Loaded from durable store (in-memory memory was empty)
After check: Paid or Failed | WorkItem = Completed (durably updated)
| Final state | PaymentConfirmed | RequiresStatusCheck | IsFallback | TransactionId | WorkItem |
|---|---|---|---|---|---|
Paid |
true | false | false | real provider id (e.g. TX-PAY-001) | Completed |
Failed |
false | false | false | null | Completed |
💡 A transaction id is earned, never invented. It appears only after the provider reports a final successful status — the Day 11 rule, carried through Day 12's reconciliation and now across a restart. Failed payments carry no id; nothing is fabricated locally.
7. Creation attempts vs status checks — the ledgers still hold
Durability must not weaken any Day 11/12 invariant. Two separate ledgers keep creation attempts and status checks apart, and the restart must add nothing to the first:
// Correct — a status READ of the original attempt, after restart:
CheckPaymentStatus("PAY-001")
// Wrong — a brand-new charge wearing the old key's name:
CreatePayment("PAY-001")
- Exactly one creation attempt per key, ever — 6 for 6 keys, all in Instance 1. Zero new creation attempts after restart.
- Six status checks after restart — one per reloaded work item, all in Instance 2.
- The idempotency key is the payment's identity; it is preserved through the durable store, so the reloaded worker asks about the original attempt, never starts a new one.
💡 Why the key still matters (Days 3–5, again). Without the preserved idempotency key in the durable record, a restarted worker would have nothing to ask about — and any "retry" would become a new charge. Durability preserves the key so follow-up stays a read.
8. The real implementation — proven by running it
Day 13 is implemented as two ISessionScenario classes registered in Program.cs, same as every previous day:
-
BEFORE:
Session13InMemoryWorkLostScenario→ commandsession-13-in-memory-work-lost -
AFTER:
Session13DurableReconciliationScenario→ commandsession-13-durable-reconciliation
Everything is simulated in-process with standard .NET only — Task.Delay for the hung gateway and the status-read latency, SemaphoreSlim(2) for the bulkhead, a linked CancellationTokenSource for the 300 ms timeout, and System.Text.Json + FileStream for the durable store (write temp → Flush(flushToDisk: true) → File.Move(overwrite: true)). Two separate ApplicationInstance objects model the crash boundary; a single ExternalPaymentProvider survives it. No SQL, no EF, no broker, no scheduler, no packages.
Real BEFORE vs AFTER metrics
| Metric (real run) | BEFORE | AFTER |
|---|---|---|
| Pending before crash | 6 / 6 | 6 / 6 |
| In-memory work items before crash | 6 | 6 |
| Durably saved (payments / work items) | — / — | 6 / 6 |
| In-memory items inherited on restart | 0 | 0 |
| Work items available after restart | 0 | 6 (reloaded from disk) |
| Status checks after restart | 0 | 6 (one per key) |
| Resolved as Paid | 0 | 4 |
| Resolved as Failed | 0 | 2 |
| Still pending | 6 / 6 | 0 / 6 |
| Completed work items | 0 | 6 / 6 |
| New payment attempts after restart | 0 | 0 |
| Duplicate charge attempts | 0 | 0 |
| Fake successes | 0 | 0 |
The AFTER reconciliation pass took ~360 ms — six status checks × ~50 ms plus the durable re-writes. Remembering the promise across a restart costs six small reads and a few file writes, not a single re-charge.
Safety checks from the real AFTER report — all passing
one creation attempt per key, ever → 6 attempts / 6 keyszero new creations after restart → 0duplicate charge attempts == 0 → 0confirmed ⇔ provider reported Paid → fake successes 0tx ids only from provider, Paid only → all match
💡 Verdicts from the real reports. BEFORE: "THE WORKER KNEW WHAT TO DO — BUT THE CRASH ERASED THE WORK ❌". AFTER: "DURABLE RECONCILIATION — THE PROMISE SURVIVED THE CRASH ✅". Both scenarios exit with code
0only when every check passes.
Run it yourself
dotnet run --project tools/Wassal.SessionTests -- session-13-in-memory-work-lost
dotnet run --project tools/Wassal.SessionTests -- session-13-durable-reconciliation
Each run prints the per-payment transition table, the summary, the checks, and the verdict, then saves a timestamped report under:
Reports/Session-13/Session-13-In-Memory-Work-Lost-RunReport-<timestamp>.txt
Reports/Session-13/Session-13-Durable-Reconciliation-RunReport-<timestamp>.txt
The AFTER durable file is created in a scenario-specific temp folder and cleaned up at the end of the run — e.g. %LOCALAPPDATA%\Temp\wassal-day13-durable-<random>\reconciliation-store.json. No durable test state is written into the application's own folders.
9. Final takeaway
- A correct worker is not enough if its work lives only in memory — a crash erases the obligation.
- When a payment becomes
PendingConfirmation, persist the payment record and its reconciliation work item together, durably, with a safe atomic write. - A restarted instance must reload its work only from the durable store — no static fields carry state across the boundary.
- Our application restarts; the external provider does not. Durability is how our side remembers what the provider still knows.
- All Day 11/12 invariants still hold: one creation per key ever, status checks are reads, no duplicate charges, transaction ids only from the provider.
- This is outbox-shaped — the direction toward a Transactional Outbox — not a production database-backed outbox.
The three days, side by side:
Day 11: We do not know yet.
Day 12: We checked the original attempt and learned the result.
Day 13: Even if our application crashes, it must remember that the check is still required.
💡 In-memory work disappears. Durable work survives restarts. Pending is a promise — durability makes sure the system remembers the promise.
Six payments went pending and were persisted durably; the process crashed; a fresh instance reloaded six work items from disk and resolved them all — 6 status checks, 4 Paid, 2 Failed, 0 new charges. A later lesson can make this a real transactional outbox with a hosted worker, schedules, and retries; the invariant to carry forward is simpler: the work must survive the process that created it.
Part of the **Fundamentals of Distributed Systems* series — building Wassal, a distributed food-delivery lab, one concept at a time.*
Top comments (0)