Day 14 made the payment and its outbox message commit atomically, and Day 13 made them durable. But a durable message with two workers is a shared message — both can read it and process it. Day 15 gives each message one temporary owner: a lease.
🙏 Credit where it's due — This series is my attempt to internalize and share what I learned from Mahmoud Youssef's excellent course, Fundamentals of Distributed Systems on Udemy. The course material, structure, and topic flow are his work; the explanations, code examples, and diagrams in these articles are my own rewrite in my own words. If you find this useful, please consider taking the course — it goes much deeper than these articles can.
Where we are — after Days 13 and 14
- Day 13 — Durable Reconciliation: the pending work survives a restart — a fresh instance reloads it from durable storage. The work survives a restart.
- Day 14 — Transactional Outbox: the payment row and the outbox row commit or roll back together in one SQLite transaction. The business state and outbox row commit together.
- Day 15 (this document): in a real deployment there is more than one application instance, each running an outbox worker. Two workers can read the same pending row. Only one worker owns a message at a time — and ownership expires if the worker dies.
🎯 The one idea to take away: Durability preserves the work. Atomicity creates the work safely. Leasing decides who owns the work now. The message is durable — now we need exclusive, temporary ownership. Durable does not mean exclusively owned; pending does not mean safe for every worker to process.
1. Why a durable, atomic outbox can still be processed twice
Day 14 answered "is the message durable and consistent?" with a firm yes. But that is a different question from "who is allowed to process it right now?"
- ✅ Days 13–14 solved — durability + atomicity: the message survives restart and always commits with its payment. One row, no orphans.
-
❌ Day 15 solves — exclusive ownership: with two app instances, both workers
SELECT … WHERE Status='Pending'and both process the same rows.
A single-worker deployment hides this. Scale to two instances and the naive worker loop becomes a race: Worker A reads the pending rows, Worker B reads the same pending rows, and both start processing before either marks anything done.
⚠️ For payment reconciliation the work is a status read, so a duplicate does not double-charge. But it is duplicate provider load, wasted work, competing DB updates, and misleading metrics — and for other message types (send an email, ship an order) a duplicate is a real, visible side effect.
2. The BEFORE scenario — competing workers, and why the final state hides it
session-15-competing-workers-no-claim seeds the Day 14 durable outbox (6 payments + 6 pending messages), then starts two workers, each on its own SQLite connection (separate application instances). A Barrier(2) makes the race deterministic — neither worker processes until both have loaded the full pending snapshot:
-
Load:
Worker-Aloads all 6 pending rows.Worker-Bloads all 6 pending rows. Neither has marked anything processed yet. -
Process: both workers call
CheckPaymentStatusfor every message they loaded — 12 processing attempts, 12 provider status checks for 6 unique messages. -
Final DB: all 6 rows end
Processed, 4 Paid + 2 Failed, no duplicate rows. The database looks perfect.
That tidy final state is the trap. From the real run: unique messages 6, Worker-A loaded 6, Worker-B loaded 6, processing attempts 12, status checks 12, messages processed more than once 6. The doubling is only visible because the report keeps the ledgers separate.
⚠️ Verdict from the real report: "THE OUTBOX WAS DURABLE — BUT TWO WORKERS PROCESSED THE SAME WORK ❌". Final row state alone cannot tell you whether work happened once or twice.
3. The lease columns and message states
The AFTER scenario adds two columns and a third status. A message now moves through:
Pending ──claim──▶ Processing ──finalize──▶ Processed
│
└── lease expires ──▶ claimable again
| New column | Meaning |
|---|---|
LockedBy |
The worker id that currently owns the message (NULL when not leased). |
LockedUntilUtc |
When the lease expires. After this time another worker may reclaim the message. |
Status is now Pending | Processing | Processed. A message is claimable when it is Pending, or Processing with an expired lease.
4. The atomic claim — and claim vs processing
The claim must be atomic, or the race just moves from "process" to "claim". A worker claims inside one BEGIN IMMEDIATE transaction (SQLite takes the write lock up front, serializing claimers) with a conditional update:
BEGIN IMMEDIATE;
SELECT Id FROM OutboxMessages
WHERE Status='Pending' OR (Status='Processing' AND LockedUntilUtc < $now)
ORDER BY CreatedAtUtc LIMIT 1;
UPDATE OutboxMessages
SET Status='Processing', LockedBy=$worker, LockedUntilUtc=$leaseUntil
WHERE Id=$id
AND (Status='Pending' OR (Status='Processing' AND LockedUntilUtc < $now));
COMMIT; -- claimed only if this UPDATE affected exactly one row
The two operations are deliberately different, and the implementation keeps two separate ledgers for them:
- Claim = acquire temporary ownership of a row. Cheap, atomic, may fail (someone else won).
-
Processing = the actual work (
CheckPaymentStatus, apply outcome). Happens only for messages you successfully claimed.
💡 Claim attempts ≠ work done. In the real AFTER run there were 5 successful claims in the concurrent phase plus a handful of empty claims (nothing claimable at that instant) — but exactly 6 unique messages were processed, once each.
5. Why leases expire — a lease is not a lock
A permanent lock would be a bug waiting to happen: if the owner crashes mid-work, the message is locked forever and the work is lost. A lease is ownership with an expiry.
- 🔒 A lock says — permanent: "Nobody else may ever take this work." If the owner dies, the work is stuck forever.
- 🎟️ A lease says — temporary: "I own this work until a specific time. If I disappear, another worker may recover it after expiry."
⚠️ Why the lease matters. A worker may claim a message and then crash. A permanent lock would lose the work forever. An expired lease lets another worker recover it — which is exactly what the recovery sub-scenario below proves.
6. Worker crash after claim, then reclaim by another worker
The AFTER scenario includes a small deterministic recovery demo on a scenario-controlled clock (so lease expiry needs no real waiting). Real run, verbatim from the report:
-
t = 09:00:00.000Z:
Worker-AclaimsPAY-001→ CLAIMED (LockedBy=Worker-A, lease 500 ms). -
crash:
Worker-Acrashes before processing — the row staysStatus=Processing, LockedBy=Worker-A. -
before expiry:
Worker-Btries to claimPAY-001→ BLOCKED (owner still Worker-A, lease not expired). -
t = 09:00:00.600Z: advance the clock past the lease →
Worker-BRECLAIMSPAY-001(LockedBy=Worker-B). -
process:
Worker-BprocessesPAY-001exactly once → outbox=Processed. The work was never lost.
A lock would have left PAY-001 stuck under a dead owner. The lease let a healthy worker take over — safely, and only after the previous owner's claim demonstrably expired.
7. Ownership-verified finalization
Claiming is only half the guarantee — completing must also check ownership, so a worker never finalizes a message it no longer owns (e.g. one whose lease expired and was reclaimed). Payment finalization and message completion happen in one transaction, guarded by LockedBy:
BEGIN IMMEDIATE;
UPDATE Payments SET PaymentStatus=…, TransactionId=… WHERE IdempotencyKey=$key;
UPDATE OutboxMessages
SET Status='Processed', ProcessedAtUtc=$now, LockedUntilUtc=NULL
WHERE Id=$id AND Status='Processing' AND LockedBy=$worker; -- ownership check
COMMIT; -- only if that UPDATE affected exactly one row; otherwise ROLLBACK
💡 Ownership violations = 0 in the real run. If the guarded update affects zero rows, the worker does not own the message any more and the transaction rolls back — it never writes a result it has no right to write.
8. The real implementation — proven by running it
Day 15 reuses the Day 14 SQLite infrastructure (OutboxSqliteCore) unchanged and adds a small leasing core plus two scenarios registered in Program.cs:
-
BEFORE:
Session15CompetingWorkersNoClaimScenario→ commandsession-15-competing-workers-no-claim -
AFTER:
Session15OutboxLeasingScenario→ commandsession-15-outbox-leasing
No new packages — the same Microsoft.Data.Sqlite. Standard .NET only: a Barrier(2) to make the BEFORE race deterministic, two SqliteConnections (WAL + busy_timeout) for two application instances, BeginTransaction(deferred:false) for BEGIN IMMEDIATE atomic claims, and a small ScenarioClock so lease expiry is deterministic without real waiting.
Real BEFORE vs AFTER metrics
| Metric (real run) | BEFORE (no claim) | AFTER (leasing) |
|---|---|---|
| Payments / pending outbox seeded | 6 / 6 | 6 / 6 |
| Worker-A loaded / Worker-B loaded | 6 / 6 (both, all rows) | claimed per-message, not bulk-loaded |
| Processing attempts | 12 | 6 |
| Unique messages processed | 6 | 6 |
| Messages processed more than once | 6 | 0 |
| Provider status-check calls | 12 | 6 |
| Ownership violations | n/a (no ownership) | 0 |
| Worker ownership split (A / B) | both did all 6 | 3 / 3 (varies by who wins each claim) |
| Resolved Paid / Failed / pending | 4 / 2 / 0 | 4 / 2 / 0 |
| Outbox Processed / Pending / Processing | 6 / 0 / 0 | 6 / 0 / 0 |
| Duplicate payment / outbox rows | 0 / 0 | 0 / 0 |
| Payment creation calls in workers | 0 | 0 |
| Fake successes | 0 | 0 |
The headline is the halving: 12 → 6 processing attempts and status checks for the same 6 messages. Same durable outcome, half the provider load, zero duplicate work. The A/B split is whatever the atomic claims happen to produce (3/3 in this run; 2/4 in another) — the invariant is 6 unique, once each.
Lease-expiry recovery proof (AFTER)
From the real report: Worker-A claimed PAY-001 then crashed → Worker-B blocked before expiry (true) → advance clock → Worker-B reclaimed after expiry (true) → reclaimed message processed once. A lock would have lost PAY-001; the lease recovered it.
Safety checks from the real AFTER report — all passing
each message processed exactly once → 6 unique, 0 duplicateszero ownership violations → 0no duplicate payment/outbox rows → max 1 per keyone gateway creation attempt per key → 6 / 6tx ids only from provider, Paid only → 4 Paid w/ id, 0 Failed w/ id
💡 Verdicts from the real reports. BEFORE: "THE OUTBOX WAS DURABLE — BUT TWO WORKERS PROCESSED THE SAME WORK ❌". AFTER: "OUTBOX LEASING — ONE OWNER PER MESSAGE, AND OWNERSHIP EXPIRES ✅". Both exit with code
0only when every check passes.
Run it yourself
dotnet run --project tools/Wassal.SessionTests -- session-15-competing-workers-no-claim
dotnet run --project tools/Wassal.SessionTests -- session-15-outbox-leasing
Each run prints the schema, the worker/claim timeline, the lease-expiry demo, the per-message ownership table, the ledgers, the checks, and the verdict, then saves a timestamped report under:
Reports/Session-15/Session-15-Competing-Workers-No-Claim-RunReport-<timestamp>.txt
Reports/Session-15/Session-15-Outbox-Leasing-RunReport-<timestamp>.txt
The SQLite file is created in a per-scenario temp folder — e.g. %LOCALAPPDATA%\Temp\wassal-day15-leasing-<random>\wassal-outbox.db — and cleaned up at the end; its path is printed in the report.
9. Final takeaway
- A durable, atomic outbox is still a shared outbox — two workers can read and process the same pending row.
- Final row state cannot prove single processing; you must count claim and processing attempts separately.
- An atomic claim (
BEGIN IMMEDIATE+ conditional update) gives exactly one worker temporary ownership of a message. - Finalization must verify ownership (
LockedBy = me) so a worker never completes a message it no longer owns. - Ownership is a lease, not a lock: it expires, so a crashed owner's work can be reclaimed instead of lost forever.
- All Day 14 guarantees still hold: one payment per key, one outbox row per payment, no orphans, no duplicate charges, provider-only transaction ids.
The three days, side by side:
Day 13: The work survives a restart.
Day 14: The business state and outbox row commit together.
Day 15: Only one worker owns a message at a time — and ownership expires if the worker dies.
💡 Durability preserves the work. Atomicity creates it safely. Leasing decides who owns it now. Retry backoff, dead-letter handling, and a hosted worker are later lessons — Day 15's invariant is exactly one temporary owner per message.
Part of the **Fundamentals of Distributed Systems* series — building Wassal, a distributed food-delivery lab, one concept at a time.*
Top comments (0)