Design a Payment System — Idempotency, Ledgers & Exactly-Once at Scale (with Production .NET Code)

"Design a payment system" is the one where scale is not the point. Nobody cares that it does ten thousand transactions a second; they care that it never does one transaction twice, never loses one, and can prove — line by line, months later — exactly where every cent went. The interview lives in four words: idempotency, ledger, and reconciliation.

This is the condensed walkthrough; the full guide (estimates, API, data model, and the full production .NET 9 code) is on my site 👇

Full guide: https://prepstack.co.in/blog/design-a-payment-system-system-design

The design at a glance

Concern	Decision
Double-charge safety	Idempotency key (unique) — a retry returns the first result, never re-charges
Money records	Double-entry ledger — balanced entries; balances derived, not stored
Provider events	At-least-once + dedup by event id (exactly-once is a myth)
Consistency	Ledger writes atomic (DB transaction); external charge bridged by a state machine
Safety net	Reconciliation — compare ledger to provider daily; flag discrepancies
Card data	Never store it — tokenize via the provider (PCI)

It's a correctness problem, not a scale problem

Say the quiet part out loud: even a large processor does a few million transactions/day ≈ tens–hundreds/sec. That fits comfortably on a well-indexed relational database. So you spend the budget not on sharding and caching but on invariants: uniqueness constraints for idempotency, a balanced ledger, and a reconciliation job. Correctness is the scarce resource here, not QPS.

Client --> [ Payment API ] -- idempotency check --> [ Provider (Stripe) ]
                |  (pending)                              |
                v                                         | webhook (at-least-once)
          [ Ledger (double-entry, atomic) ] <-- dedup ---+  payment_intent.succeeded
                |
          [ Reconciliation job ] -- daily compare vs provider --> flag discrepancies

The hard parts

Idempotency — the whole ballgame. A client charges, the request times out, the client retries. Did the first attempt succeed? You can't know from the client side — so the server makes the retry safe. The client sends an idempotency key; the server stores it under a unique constraint before doing anything expensive. On a retry the key already exists, so you return the stored result instead of charging again. Pass the same key to the provider too (Stripe's Idempotency-Key header) so even the external call dedups. One key, one charge, forever.

You can't make the charge and the ledger one transaction. The provider is external — you cannot enroll "call Stripe" and "write my ledger" in one ACID transaction. Use a state machine: record the payment as pending (committed), call the provider, and let the webhook drive the authoritative transition to succeeded/failed, at which point you post the ledger — all in one local transaction. Internal ledger strongly consistent; external settlement eventually consistent; the webhook is the bridge.

Double-entry ledger — the books must balance. Never store a single mutable balance. Every money movement posts balanced entries — debits equal credits, netting to zero — into an immutable, append-only ledger. A successful charge might debit Cash and credit the customer's Receivable. Balances are derived (SUM over an account), so the ledger is auditable, replayable, and self-checking: if a transaction's entries don't sum to zero, reject it.

Webhooks are at-least-once — dedup them. The provider will occasionally deliver the same payment_intent.succeeded twice; applying it twice double-credits the ledger. Guard every handler with a UNIQUE(eventId) dedup log: record the event id first; if it's already there, the second delivery is a no-op.

Reconciliation — the backstop. What if a webhook is lost? The charge succeeded at the provider but your ledger never recorded it — a silent gap. A scheduled job pulls the provider's transaction list and compares it to your ledger, flagging anything that exists on one side but not the other. Reconciliation turns "hope the webhook arrived" into "prove the books match."

Failure & scaling gotchas

• Provider timeout (the classic): you called Stripe and got no response. Don't blindly retry the charge — retry with the same idempotency key (safe), or query the provider for the intent's status. Reconciliation catches whatever slips through.
• Ledger scale: append-only writes scale well; partition by account/time; balances via periodic snapshots + entries since.
• Outbox for downstream events: publish payment.succeeded to other services via the outbox pattern so a crash can't drop the event.
• Refunds/reversals are just more balanced entries — never delete or mutate existing ones.
• PCI: never let raw card numbers touch your servers; tokenize through the provider and store only the token.

I shipped this in production (Mattrx)

Mattrx bills ~thousands of tenant workspaces through Stripe. V1 was dangerously naive: it charged inline, and a timeout on Stripe's response left it not knowing whether the charge landed — so a retry occasionally double-charged a customer, cleaned up by hand. "Paid" was a boolean on the subscription row, so there was no audit trail and monthly reconciliation was a spreadsheet. We rebuilt it as exactly the design above:

Metric	Before	After
Double-charges on retry	A handful a month (manual refunds)	0 (idempotency key + Stripe Idempotency-Key)
Money record	paid boolean on the subscription	Immutable double-entry ledger (always reconciles)
Webhook double-application	Occasional double-credit	0 (UNIQUE(eventId) dedup)
Reconciliation vs Stripe	Manual spreadsheet (~hours/month)	Automated nightly, discrepancies auto-flagged
Billing disputes from double-charges	Non-zero	0 in the last two quarters

The charge path reserves the idempotency key before calling Stripe and passes the same key onward, so a client retry and a provider retry both collapse to one charge; the webhook handler dedups on event_id and posts a balanced ledger transaction inside one local DB transaction; and Ledger.PostAsync refuses to write anything that doesn't net to zero — so the books are correct by construction, and the nightly reconciliation job catches the one thing code can't: a settlement Stripe recorded but never told us about. (Full .NET 9 billing service + webhook handler + ledger is in the post.)

The model to carry forward

A payment system is a machine for staying correct when you don't know what happened. The timeout is the enemy, and every part of the design is a defense against it: an idempotency key so retries are safe, a state machine so an external charge and an internal record don't have to be one transaction, a double-entry ledger so the truth is auditable and self-checking, deduped webhooks so at-least-once behaves like once, and reconciliation so a lost message can't quietly corrupt the books.

Three habits it teaches: lead with idempotency ("the client sends an idempotency key" frames the entire correctness story); make money a ledger, not a number (immutable balanced entries with derived balances is the only design an auditor or a bug can't break); assume the webhook can be lost (reconciliation is what turns hope into proof).

The full guide has the estimates, API, data model, all the hard parts in depth, failure/scaling, the complete production .NET 9 code, and the "when it's overkill" section:

https://prepstack.co.in/blog/design-a-payment-system-system-design

Originally published on PrepStack.