Day 12 — Payment Reconciliation: Resolving the Unknown Safely

Day 11 taught the honest fallback: when the provider times out, say "we do not know yet". But pending-confirmation is a promise that someone will check again. Day 12 keeps that promise — with a reconciliation worker that asks about the original attempt, and never, ever charges again.

🙏 Credit where it's due — This series is my attempt to internalize and share what I learned from Mahmoud Youssef's excellent course, Fundamentals of Distributed Systems on Udemy. The course material, structure, and topic flow are his work; the explanations, code examples, and diagrams in these articles are my own rewrite in my own words. If you find this useful, please consider taking the course — it goes much deeper than these articles can.

🎯 The one idea to take away. Fallback admits uncertainty. Reconciliation resolves uncertainty. Day 11: "We do not know yet." — Day 12: "We checked the original attempt and now know the final result." Resolving means asking about the original attempt — never charging again.

Where we are — after Day 11

• Day 11 — Payment Fallback Without Lying: when the payment provider times out, we return an honest degraded response — PendingConfirmation, PaymentConfirmed = false, IsFallback = true, RequiresStatusCheck = true, idempotency key preserved, no fabricated transaction id.
• Day 12 (this document): that response contains a to-do item — RequiresStatusCheck = true. Who performs the check? How does a payment move from pending to a real final state? And how do we prove the check never becomes a second charge?

1. Why PendingConfirmation cannot remain forever

PendingConfirmation is honest — and it is also not a final business outcome. Nothing downstream can act on it:

• The customer who was charged never gets an order confirmation — money taken, nothing delivered.
• The customer who was not charged is never told to retry — the sale is silently lost.
• Finance cannot close the books on a payment that is neither paid nor failed.

• Support sees "pending" forever and has no answer to give.

• ✅ Honest fallback (Day 11 · solved) — On timeout, say the truth: pending, not confirmed, check later. Nobody is lied to.

• ❌ The promise-keeper (Day 12 · still missing) — RequiresStatusCheck = true is a to-do with no owner. Nothing checks, so nothing ever resolves.

⚠️ Honest but incomplete. An unresolved pending payment is a debt of truth. The fallback bought time honestly — reconciliation is how the system pays the debt back.

2. The BEFORE problem — nobody ever checks

The BEFORE scenario (session-12-no-reconciliation) replays Day 11 exactly, then lets time pass:

• Phase 1: 6 payments (PAY-001…PAY-006) hit a hung gateway (2000 ms) behind the Day 9 bulkhead with the Day 10 timeout (300 ms). All 6 time out → all 6 become PendingConfirmation, each stored with its idempotency key.
• Phase 2: The provider recovers. It now knows the real fate of every original attempt: 4 were actually charged, 2 actually failed. The answers exist and are one status call away.
• …forever: Our system has no reconciliation worker. Status checks performed: 0. Resolved: 0. All 6 payments stay pending until the end of time.

From the real run: initial pending 6, provider recovered true, status checks 0, resolved 0, still pending 6, duplicate charges 0.

⚠️ Verdict from the real report: "THE SYSTEM ADMITTED UNCERTAINTY — BUT NEVER RESOLVED IT ❌". Every check in the BEFORE report passes — because the problem it proves is real: honesty without follow-up leaves the business stuck.

3. The AFTER fix — a reconciliation worker

The AFTER scenario (session-12-background-reconciliation) is identical through Phase 1 and the provider recovery. Then one new piece runs — a small, deterministic, in-process reconciliation worker:

1. Scan the pending store — only payments still in PendingConfirmation are work.
2. For each, call CheckPaymentStatus(key) — a status read of the original attempt, identified by the same idempotency key. A ~50 ms read, not a charge.
3. Apply the provider's real answer: Paid → confirmed with the provider's transaction id; Failed → a real failed state, no id.
4. Clear RequiresStatusCheck — the promise is kept, the payment is final.

From the real run, every payment resolved to the provider's truth:

PAY-001 | PendingConfirmation → Paid
PAY-002 | PendingConfirmation → Paid
PAY-003 | PendingConfirmation → Failed
PAY-004 | PendingConfirmation → Paid
PAY-005 | PendingConfirmation → Failed
PAY-006 | PendingConfirmation → Paid

💡 Kept deliberately small. The worker is a plain awaited loop — no hosted service, no queue, no scheduler, no database. Durability, crash recovery, and retry schedules belong to later lessons. Day 12's invariant is the part worth keeping forever: ask, don't re-charge.

4. Checking a payment is not retrying a payment

This is the sharpest edge of the lesson. Two operations that sound similar and must never be confused:

// Correct — a status READ of the original attempt:
CheckPaymentStatus("PAY-001")

// Wrong — a brand-new charge wearing the old key's name:
CreateNewPayment("PAY-001")

• Same idempotency key = same original payment attempt. The key is the payment's identity, carried from Day 11's fallback into Day 12's check.
• A status check asks: "what happened to attempt PAY-001?" It moves no money.
• A new creation says: "charge the customer." If the original attempt actually succeeded at the provider, this is a double charge.
• That is why the implementation keeps two separate ledgers — payment creation attempts and status-check calls — and the report never hides them behind one generic "provider calls" counter.

⚠️ Why the key matters (Days 3–5, again). Without the preserved idempotency key, "check later" is impossible — there is nothing to ask about — and any retry becomes a new charge. The key is what turns follow-up from a gamble into a read.

5. Explicit state transitions

Three states, and every transition is driven by the provider's reported truth — never by our hopes:

PendingConfirmation ──(provider says Paid)───→ Paid
PendingConfirmation ──(provider says Failed)─→ Failed
PendingConfirmation ──(still unresolved)─────→ PendingConfirmation  (check again later)

Final state	PaymentConfirmed	RequiresStatusCheck	IsFallback	TransactionId
Paid	true	false	false	real provider id (e.g. TX-PAY-001)
Failed	false	false	false	null
PendingConfirmation (unresolved)	false	true	true	null

💡 A transaction id is earned, never invented. It appears only after the provider reports a final successful status — exactly the Day 11 rule, carried forward: no fake success, no fabricated id, and a provider-reported failure becomes a real Failed, not a quiet retry.

6. The real implementation — proven by running it

Day 12 is implemented as two ISessionScenario classes registered in Program.cs, same as every previous day:

• BEFORE: Session12NoReconciliationScenario → command session-12-no-reconciliation
• AFTER: Session12BackgroundReconciliationScenario → command session-12-background-reconciliation

Everything is simulated in-process with standard .NET primitives — Task.Delay for the hung gateway and the status-read latency, SemaphoreSlim(2) for the bulkhead, a linked CancellationTokenSource for the 300 ms timeout, and an in-memory dictionary of provider outcomes (deterministic: 4 Paid with transaction ids, 2 Failed). Two ConcurrentDictionary ledgers count creation attempts and status checks per key, so the safety claims are measured, not asserted.

Real BEFORE vs AFTER metrics

Metric (real run)	BEFORE	AFTER
Initially pending	6 / 6	6 / 6
Provider later recovered	true	true
Payment creation attempts (ever)	6 (6 keys)	6 (6 keys)
Creation attempts during reconciliation	—	0
Status checks performed	0	6 (one per key)
Resolved as Paid	0	4
Resolved as Failed	0	2
Still pending	6 / 6	0 / 6
Duplicate charge attempts	0	0
Fake successes	0	0
Total scenario duration	938 ms	1250 ms

The AFTER run is ~300 ms longer — that is the reconciliation pass itself: 6 status checks × ~50 ms. Resolving the unknown costs six small reads, not one risky re-charge.

Safety checks from the real AFTER report — all passing

• one creation attempt per key, ever → 6 attempts / 6 keys
• zero creations during reconciliation → 0
• duplicate charge attempts == 0 → 0
• confirmed ⇔ provider reported Paid → fake successes 0
• transaction ids only from provider → all match
• failed payments carry no tx id → none do

💡 Verdicts from the real reports. BEFORE: "THE SYSTEM ADMITTED UNCERTAINTY — BUT NEVER RESOLVED IT ❌". AFTER: "BACKGROUND RECONCILIATION — THE UNKNOWN WAS RESOLVED, AND NOBODY WAS CHARGED TWICE ✅". Both scenarios exit with code 0 only when every check passes.

Run it yourself

dotnet run --project tools/Wassal.SessionTests -- session-12-no-reconciliation
dotnet run --project tools/Wassal.SessionTests -- session-12-background-reconciliation

Each run prints the per-payment transition table, the summary, the checks, and the verdict, then saves a timestamped report under:

Reports/Session-12/Session-12-No-Reconciliation-RunReport-<timestamp>.txt
Reports/Session-12/Session-12-Background-Reconciliation-RunReport-<timestamp>.txt

7. Final takeaway

• PendingConfirmation is honest, but it is not a final business outcome — it is a promise to check again.
• Reconciliation keeps the promise by asking about the original attempt — CheckPaymentStatus(key), never CreateNewPayment(key).
• The same idempotency key means the same payment attempt. One creation per key, ever; status checks are reads.
• A payment becomes Paid only when the provider reports it; a provider-reported failure becomes a real Failed; an unresolved answer may stay pending — to be checked again.
• Transaction ids come from the provider or not at all.

The two days, side by side:

Day 11:  We do not know yet.
Day 12:  We checked the original attempt and now know the final result.

💡 Fallback admits uncertainty. Reconciliation resolves uncertainty. Together they let a payment system survive a provider outage without ever lying — and without ever charging twice.

Pending is a promise to check again. Day 12 keeps the promise. Six payments went pending honestly, the provider recovered, and one small worker resolved them all — 6 status checks, 4 Paid, 2 Failed, 0 new charges. A later lesson can make the worker durable (schedules, queues, crash recovery); the invariant to carry forward is simpler: ask, don't re-charge.

---

Part of the **Fundamentals of Distributed Systems* series — building Wassal, a distributed food-delivery lab, one concept at a time.*