System design case study: Stripe and payment systems

Part 7 of 8 · week 7 of the learning path

In a payment system the hard problem is correctness, not scale: never charge twice, never lose a record, and always be able to prove where every cent went.

Learning objectives

• Design idempotent APIs and a payment state machine.
• Explain double-entry ledgers, reconciliation and why timeouts are unknown outcomes.
• Design reliable, signed webhooks and keep card data out of most of the system.

1. Requirements

• Functional: create and confirm payments, authorize then capture, refunds, support for cards and wallets, extra authentication steps such as 3D Secure, webhooks to merchants, payouts to merchant bank accounts, fraud checks and reports.
• Non-functional: exactly-once effect on money, strong consistency for balances, durability, a full audit trail, high availability, low latency for checkout, and compliance with card data security rules (PCI DSS).

2. Estimates. Assume 100 million payments a day; the aim is to show that correctness is harder than volume.

Quantity	Calculation	Result
Average payments per second	100 million divided by 86,400	about 1,200
Peak at 3 times average	1,200 times 3	about 3,500; a shopping holiday can reach several times that
Ledger and event storage at about 5 KB per payment	100 million times 5 KB	about 500 GB per day, about 180 TB per year
Webhook events at 3 per payment	300 million per day	about 3,500 per second, plus retries
Downtime budget at 99.99%	per year	about 53 minutes, each minute costs merchants sales

3. API and data model

• API: create a payment with an idempotency key, confirm it, capture it, refund it, and list events. Merchants receive webhooks at their own endpoint.
• Data: payments (state and version), attempts, ledger entries, idempotency keys with the stored response, outbox events, merchants and payouts. Money is stored as integers in the smallest unit plus a currency code, never as floating point.

4. High-level design

 Merchant app
     | (1) POST /payments + idempotency key
     v
 API gateway (auth, idempotency check)
     | (2)
     v
 Payment service (state machine) --(3)--> Risk check
     | (4) authorize                           
     v
 Card network
     |
     v (5) one transaction
 Ledger + outbox
     | (6) publish from outbox
     v
 Webhook sender --signed webhook--> Merchant (7)

 Reconciliation job: compares ledger with settlement files from the network

Step numbers in the diagram match the list below.

The numbers match the steps below; the highlighted payment service owns the state machine and the single transaction that makes a payment real.

1. The merchant's server creates a payment, sending an idempotency key.
2. The API gateway authenticates, rate limits and checks the key, then forwards to the payment service.
3. The payment service asks the risk check for a score before going further.
4. It asks the card network, through an acquirer, to authorize the payment.
5. It records the ledger entries and an outbox event in one database transaction.
6. The outbox event goes to the webhook sender.
7. The webhook sender delivers a signed event to the merchant, retrying until it is acknowledged.

5. Deep dives

• Idempotency: the first request with a key inserts it in a unique index and runs; a retry returns the stored response; the same key with different parameters is rejected; concurrent duplicates wait or get a conflict. Keys are kept for about a day.
• State machine: only legal transitions are allowed (created, requires action, processing, authorized, succeeded, failed, canceled, refunded). Updates use a version number so two concurrent updates cannot both win, and final states never change.
• Unknown outcomes: if the card network times out, the payment is neither failed nor succeeded. Mark it processing, query its status or wait for the asynchronous result, and never blindly retry the authorization. A nightly reconciliation catches whatever slips through.
• Ledger: double-entry and append-only. Every transaction posts entries whose debits equal credits, corrections are new reversing entries, and the sum of all entries must always be zero. Use a relational database with strong isolation, sharded by merchant account.
• Card data and security: card details go from the customer's browser straight to an isolated vault that returns a token, so the rest of the system never sees card numbers and its compliance scope stays small. Keys live in hardware security modules and every access is audited.
• Fraud and risk: a rules and model score computed within a tight time budget before authorization decides to allow, challenge with extra authentication, or block. Decide up front whether the system fails open or closed if the risk service is down.
• Webhooks: delivered at least once from the outbox, signed with a secret and a timestamp so merchants can check authenticity and reject replays, and retried with exponential backoff for days. Merchants must dedupe by event ID and tolerate out-of-order events by fetching the current state.
• Payouts and reconciliation: batch jobs pay merchants out, and reconciliation compares the ledger with the settlement files from networks and banks, sending mismatches to a review queue.
• API evolution: pin each merchant to a dated API version so changes never break integrations unexpectedly.

6. Trade-offs: strong consistency over availability for money; synchronous confirmation is simpler for merchants but couples them to slow card networks, so asynchronous confirmation with webhooks is more robust; a single home region per payment is simpler and safer than active-active writes.

7. Failures and monitoring. On a database failover, idempotency keys and the state machine keep retries safe. A slow card network triggers timeouts and a circuit breaker, with traffic rerouted to another acquirer. Watch authorization success rate by acquirer, p99 latency, the ledger imbalance (which must be zero, and pages someone if not), reconciliation mismatches, webhook delay and the idempotency conflict rate.

Real-world notes: Stripe publicly documents idempotency keys, signed webhooks with automatic retries for several days, and date-based API versions; these are the same patterns described above.

Design exercise: add marketplace payouts, where a platform takes a fee and splits each payment between sellers. Show the ledger entries for one 100 dollar sale with a 10 dollar platform fee, and say what happens if a refund arrives after the seller has been paid out.

Quiz 7 (5 questions, pass mark 4)

1. A merchant's request times out and the client retries. What prevents a double charge?
   • A) A faster network
   • B) An idempotency key that makes the retry return the original result
   • C) A bigger database
   • D) Sending a new key on every retry
2. Why store money as integers in minor units?
   • A) Integers use less memory only
   • B) Currencies have no decimals
   • C) Databases cannot store decimals
   • D) Floating point cannot represent many decimal amounts exactly, causing rounding errors
3. What is true of a double-entry ledger?
   • A) Every transaction's debits equal its credits, and entries are append-only
   • B) Balances are edited in place
   • C) It records only credits
   • D) It is optional for payment systems
4. The card network times out on an authorization. What should the system treat it as?
   • A) A definite failure, so retry at once
   • B) A definite success
   • C) An unknown outcome to be checked by status query and reconciliation
   • D) A reason to delete the payment
5. Why must a webhook handler be idempotent?
   • A) Webhooks are never repeated
   • B) Delivery is at least once with retries, so an event can arrive more than once or out of order
   • C) Merchants have only one server
   • D) Signatures are optional

Quiz 7 answers

1. B an idempotency key makes a retry return the stored result instead of charging again.
2. D
3. A
4. C
5. B

Sources and further reading

• Stripe: idempotent requests, on keys, stored responses and pruning after at least 24 hours.
• Stripe: webhooks, on events, signatures and retries.
• Stripe: how API versioning works, on versions pinned per account and per request.
• Ledger: Stripe's system for tracking and validating money movement, Stripe Engineering, 2024.
• PCI Security Standards Council, the owner of PCI DSS.

The ledger, state machine and reconciliation design is a teaching design in the spirit of these sources, not Stripe's internal architecture.

---

Part 7 of 8 in the System Design Learning Path series. Next: part 8.