Day 11 — Payment Fallback: Without Lying

Bulkhead limits how many. Circuit breaker stops the storm. Timeout stops the wait. But when the payment provider still cannot give a final answer, one business question remains: what do we tell the caller? For payments, the fallback must be honest — never a fake success.

🙏 Credit where it's due — This series is my attempt to internalize and share what I learned from Mahmoud Youssef's excellent course, Fundamentals of Distributed Systems on Udemy. The course material, structure, and topic flow are his work; the explanations, code examples, and diagrams in these articles are my own rewrite in my own words. If you find this useful, please consider taking the course — it goes much deeper than these articles can.

🎯 The one idea to take away. The safest fallback is not a fake answer. It is an honest state. For payments, unknown is a valid state. Do not convert unknown into success — and do not convert it into failure either. Say what is true: we do not know yet.

Where we are — after Day 10

• Day 8 — Circuit Breaker: when a dependency keeps failing, stop hammering it. Fail fast and give it room to recover.
• Day 9 — Bulkhead Isolation: give each dependency its own pool of slots, so one slow dependency cannot starve the others.
• Day 10 — Timeout & Cancellation: bound how long a call may hold a slot. Stop waiting, cancel, release.
• Day 11 (this document): all of that protects the system. Now we answer for the caller: what should the system return when the dependency still cannot give a final answer?

1. Three protections in place — one question still open

After Days 8–10, our calls to a payment provider are well protected. Each mechanism answers its own question:

• ✅ Bulkhead (Day 9 · how many?) — At most N payment calls run at once. One slow dependency cannot starve the rest.
• ✅ Circuit breaker (Day 8 · keep trying?) — When the provider keeps failing, stop hammering it and fail fast for a while.
• ✅ Timeout (Day 10 · how long?) — A call may wait ~300 ms, then it is cancelled and its slot is released.
• ❌ Fallback (Day 11 · what to return?) — The timeout fired. The breaker is open. What do we tell the caller? Nothing so far decides that.

⚠️ The gap. Timeout decides when to stop waiting. It does not decide what to say after you stopped. All three mechanisms are infrastructure answers — the fallback is a business answer, and it must match the business risk.

2. Why a "normal" fallback is not safe for payments

For a read-only, low-risk feature, a fallback can be simple and nobody gets hurt. Product recommendations are the classic example:

• return cached recommendations,
• return the popular items list,
• or even return an empty list.

Worst case, the customer sees slightly generic suggestions. The business risk is near zero — so a cheap default is a perfectly good fallback.

Payments are the opposite end of the risk scale. A payment call moves money and changes state. And a payment timeout is ambiguous by nature: the provider may have already charged the card — we simply never received the confirmation. There is no "safe default value" for money.

Question	Recommendations	Payments
What does the call do?	Reads data	Moves money, changes state
Is a default value harmless?	Yes — popular items are fine	No — there is no default for money
What does a timeout mean?	No suggestions this time	Unknown — the charge may have happened
Cost of a wrong fallback	A slightly generic page	Double charge, broken reconciliation, lost trust
Good fallback	Cached / popular / empty	An honest pending state

⚠️ Fallback must match the business risk. The same pattern — "return something instead of an error" — is harmless for recommendations and dangerous for payments. Copying a read-side fallback onto a payment path is how systems start lying.

3. The wrong solution — fake success

The tempting shortcut: the provider timed out, the customer is waiting, so just… pretend it worked and sort it out later.

PaymentStatus = Paid
TransactionId = FAKE-123
Message       = Payment completed successfully

This response is a lie, and every part of the business eventually pays for it:

• The customer may receive an order without a real payment — the charge never happened, but the order shipped.
• Finance reconciliation breaks — FAKE-123 matches nothing at the provider. Someone has to untangle it by hand.
• Support cannot know the real payment state — the system itself no longer knows which payments are real.
• Retrying later may double-charge the customer — the original attempt may have succeeded at the provider; a "retry" creates a second real charge.
• The system loses truth — once fake data enters your records, every report, refund, and audit built on top of it inherits the lie.

⚠️ Do not convert unknown into success. A fake Paid does not remove the uncertainty — it hides it inside your own database, which is the worst place for it to live.

4. The correct solution — explicit pending confirmation

The honest answer has three properties: it does not say the payment failed, it does not say the payment succeeded, and it clearly marks itself as a degraded response that needs a follow-up.

PaymentStatus       = PendingConfirmation
PaymentConfirmed    = false
IsFallback          = true
RequiresStatusCheck = true
Message             = Payment provider did not confirm in time.
                      Payment status must be checked later.

Read it back in plain words: "We asked the provider to charge the card. We did not hear back in time. We do not know yet — and here is what to do about it." That is the honest distributed systems answer.

As a response model, it stays small and boring on purpose:

{
  "orderId": 1001,
  "paymentStatus": "PendingConfirmation",
  "paymentConfirmed": false,
  "isFallback": true,
  "requiresStatusCheck": true,
  "message": "Payment provider did not confirm in time. Please check payment status later."
}

Field	Value	Why it is there
paymentStatus	PendingConfirmation	Not success, not failure — the true state.
paymentConfirmed	false	Nothing downstream (order confirmation, shipping) may proceed as if money moved.
isFallback	true	The response is clearly marked as degraded — the UI and the logs can both tell.
requiresStatusCheck	true	An actionable next step, not a dead end.
message	"…check payment status later."	Something honest the caller can show a human.

💡 For payments, unknown is a valid state. PendingConfirmation gives uncertainty a name and a home in your model — instead of forcing it, wrongly, into Paid or Failed.

5. What the caller should do next

A pending response is only useful if the caller knows what to do with it. Three simple rules:

• Show the truth to the user. "Your payment is being confirmed — we will update you shortly." Not "Payment successful", not "Payment failed".
• Do not trigger success-only side effects. No order confirmation email, no shipping label, no loyalty points — those wait for paymentConfirmed = true.
• Check status later, or retry with the same idempotency key. The follow-up asks about the same payment attempt — it never starts a new one.

💡 Kept deliberately simple. In a production system the "check later" part often becomes a background reconciliation job that polls the provider and settles pending payments automatically. That deserves its own lesson — Day 11 stays focused on the honest response itself.

6. Why idempotency still matters

The pending state only works because of an old friend from Days 3–5: the Idempotency-Key. The fallback must preserve it, because the key is what makes "try again later" safe:

• Same order + same idempotency key = same payment attempt. Always.
• A retry checks or continues that same attempt — it never creates a second one.
• Never create multiple real payment attempts for one customer action.
• The pending status must be replayable: asking again returns the same pending result, or the latest known status — not a new charge.

The idempotent retry story, in two beats:

• First request: OrderId = 1001, Idempotency-Key = abc-123 → payment provider timeout → return PaymentPendingConfirmation.
• Second request: Same Idempotency-Key = abc-123 → do not create a second payment → return the same pending result, or the latest known status.

⚠️ The double-charge trap. A retry with a new key is a brand-new payment in the provider's eyes. If the first (timed-out) attempt actually succeeded, the customer is now charged twice. The idempotency key is the difference between "asking again" and "charging again".

7. How this connects to the previous days

Each day answered one question. Day 11 adds the final, business-facing one:

Day	Mechanism	The question it answers
Days 3–5	Idempotency	How do we make retries safe?
Day 8	Circuit breaker	Should we keep trying a failing dependency?
Day 9	Bulkhead	How many calls may run at once?
Day 10	Timeout + cancellation	How long may a call wait?
Day 11	Honest fallback	What do we return when there is still no final answer?

Notice the shape: Days 8–10 protect the system and never touch the response. Day 11 protects the truth of the response — and it leans on Days 3–5 to make the follow-up safe. All of it composes: the bulkhead and timeout free the slot quickly, the breaker stops the storm, the idempotency key keeps the retry safe, and the honest fallback tells the caller exactly where things stand.

8. The real implementation — proven by running it

This lesson is not just a diagram — it is implemented and verified in the test runner as two ISessionScenario classes, registered in Program.cs like every previous day:

• BEFORE: Session11NoSafeFallbackScenario → command session-11-no-safe-fallback
• AFTER: Session11SafeFallbackScenario → command session-11-safe-fallback

How the failure is simulated

Everything runs in-process — no server, no database, no packages. The payment gateway is a bounded Task.Delay(2000 ms) that never answers in time. Each call runs through the Day 9 bulkhead (SemaphoreSlim(2)) with the Day 10 per-call timeout: a linked CancellationTokenSource with CancelAfter(300 ms). Six payment operations are fired, each carrying its own idempotency key: PAY-001 … PAY-006. In both scenarios all 6 gateway calls time out — the difference is only what the caller is told.

What BEFORE demonstrates

The timeout path returns a bare hard failure: Result = failed, Reason = payment-timeout — with no pending state, no RequiresStatusCheck, no retry advice, and the idempotency key not echoed back. Every infrastructure check passes (slots freed at ~305 ms, whole run ~920 ms), yet user-visible graceful responses are 0 / 6. The system is protected; the caller hits an ambiguous dead end.

What AFTER changes

One thing only: the catch (OperationCanceledException) block builds the honest fallback instead of a bare failure —

response = new PaymentResponse
{
    Result              = "pending-confirmation",
    PaymentConfirmed    = false,
    IsFallback          = true,
    Source              = "safe-payment-fallback",
    Reason              = "primary-timeout",
    RequiresStatusCheck = true,
    RetryAdvice         = "retry-with-same-idempotency-key",
    IdempotencyKey      = idempotencyKey,   // preserved and echoed back
    TransactionId       = null,             // never fabricated
};

A gateway-attempt ledger (a ConcurrentDictionary keyed by idempotency key) counts every real gateway call, so the report can prove there was exactly one attempt per key — no silent automatic retry.

What the real run proves

From the generated reports (both scenarios exit with code 0):

Metric (real run)	BEFORE	AFTER
Payment gateway timeouts	6 / 6	6 / 6 (still down)
Hard failures returned to caller	6 / 6	0 / 6
Pending-confirmation responses	0 / 6	6 / 6
Fallback clearly marked (IsFallback)	0 / 6	6 / 6
Fake successes / confirmed payments / fabricated tx ids	0 / 0 / 0	0 / 0 / 0
Idempotency keys preserved & echoed	0 / 6	6 / 6
RequiresStatusCheck responses	0 / 6	6 / 6
Duplicate charge attempts	0	0 (6 attempts / 6 keys)
Avg slot-hold time	305 ms	300 ms
Total scenario duration	920 ms	905 ms
User-visible graceful responses	0 / 6 (0%)	6 / 6 (100%, all degraded)

Hold time is ~300–305 ms, but average response time is higher (~600 ms) because later waves wait in the bulkhead queue first: response time = wait + hold.

The AFTER report ends with dedicated safety checks, all passing on the real run: fake successes == 0, confirmed payments == 0 (timed out), fabricated transaction ids == 0, every fallback marked IsFallback=true → 6 / 6, idempotency key preserved & echoed → 6 / 6, and exactly one gateway attempt per key → 6 attempts / 6 keys, duplicates 0.

💡 Verdicts from the real reports. BEFORE: "TIMEOUT PROTECTED THE SYSTEM — BUT EVERY PAYMENT ENDED IN AN AMBIGUOUS HARD FAILURE ❌". AFTER: "SAFE PAYMENT FALLBACK — GATEWAY STILL DOWN, NOBODY WAS LIED TO ✅". Same hung gateway, same timeout — only the honesty of the answer changed.

Run it yourself

dotnet run --project tools/Wassal.SessionTests -- session-11-no-safe-fallback
dotnet run --project tools/Wassal.SessionTests -- session-11-safe-fallback

Each run prints the per-operation table, summary, checks, and verdict, then saves a timestamped report under:

Reports/Session-11/Session-11-No-Safe-Fallback-RunReport-<timestamp>.txt
Reports/Session-11/Session-11-Safe-Fallback-RunReport-<timestamp>.txt

9. Final takeaway

• A fallback is not always "return some default data". Fallback must match the business risk.
• For low-risk reads (recommendations), a cached or popular-items default is fine.
• For payments, a timeout means unknown — and unknown is a valid state. Give it a name: PendingConfirmation.
• Never return Paid, a fake transaction id, or "payment completed" without real confirmation from the provider.
• Preserve the idempotency key so that checking or retrying later continues the same attempt instead of charging twice.

💡 Do not convert unknown into success. The system that admits "we do not know yet" keeps its truth — and a system that keeps its truth can always recover. A system that lies to its own database cannot.

The safest fallback is not a fake answer. It is an honest state. Bulkhead, breaker, and timeout protect the system. The honest fallback protects the truth. For payments that means: not success, not failure — pending, clearly marked, safely retryable. A later lesson can add background reconciliation and status checking on top of this foundation.

---

Part of the **Fundamentals of Distributed Systems* series — building Wassal, a distributed food-delivery lab, one concept at a time.*