Most systems can tolerate being a little bit wrong. A like count that is off by one, a feed that is a few seconds stale, a search result slightly out of order. Nobody gets hurt. A payment system does not have that luxury, because being wrong means charging someone twice or moving money into a void, and that is real harm to a real person. So the design goal is not just fast or scalable. It is correct under failure, and failure is guaranteed, because networks time out and servers crash in the middle of things.
The first problem to solve is duplicates. Imagine a user taps pay, the request reaches your server, you charge their card, and then the response back to their phone is lost to a flaky network. Their app, having heard nothing, retries. Now you are about to charge them a second time for one purchase. This is not a rare edge case, it is the normal behavior of unreliable networks, and any payment design that ignores it is broken. The answer is idempotency. The client attaches a unique key to the payment request, and the server records that key the first time it processes the request. If a request arrives with a key the server has already seen, it does not charge again. It returns the result of the original attempt. So a retry is safe, because the retry is recognized as the same operation, not a new one. Getting idempotency right is arguably the single most important decision in the whole system.
The second problem is knowing the truth about money, and this is where the ledger comes in. The wrong model is to keep a single balance number and mutate it up and down, because a mutable number tells you the current value but not how you got there, and when something looks wrong you have no way to reconstruct what happened. The right model is a ledger: an append only record where every movement of money is a new immutable entry. You never edit or delete an entry. To reverse a mistake you add a compensating entry. A balance is then not a stored number you trust, it is something you can derive by summing the entries. This is double entry accounting, the same idea banks have used for centuries, and it gives you an audit trail where every cent is traceable to the events that produced it. The trade-off is that you write more data and compute balances rather than reading a single field, but for money that is exactly the trade you want.
The third problem is that a payment is not one instant, it is a process with stages, and it can fail at any stage. Money gets authorized, then captured, then settled, and a payment can also be pending, failed, or refunded. Modeling this as a state machine, with explicit states and only certain allowed transitions, keeps the system honest. A payment cannot jump from failed to settled. A refund only applies to something that was actually captured. When a step fails partway, the state machine tells you exactly where you are and what recovery is valid, instead of leaving you guessing. This matters most when a payment spans more than one system, for example your service and an external card processor, because now you have two parties that can disagree about what happened.
That cross system disagreement is the fourth problem, and the honest answer is that you cannot make two independent systems update in one perfect atomic step. So you do not pretend to. You record your intent first, then call the external processor, then reconcile. If you sent a charge to the processor but never got a confirmation, you do not assume success or failure. You have a reconciliation process that later checks with the processor for the true outcome and settles your records to match. This is eventual consistency applied to money, done carefully, with the ledger and idempotency making it safe to retry and safe to verify. Accepting that you will sometimes be temporarily unsure, and building a reliable process to resolve that uncertainty, is more honest and more correct than pretending distributed atomic writes exist.
How does the real company do it? Payment platforms like PayPal and Stripe lean hard on idempotency keys so retries never double charge, an append only double entry ledger as the source of truth for balances, an explicit state machine for the lifecycle of each payment, and asynchronous reconciliation against banks and card networks to settle the truth after the fact. They favor being auditable and eventually correct over being fast and occasionally wrong.
The mindset shift for an interview is this: stop trying to make the operation instant and atomic, and instead make every step safe to retry, every movement recorded immutably, and every uncertain outcome resolvable by reconciliation. That is how you build something that never loses a cent.
I wrote the full breakdown, with diagrams and the data model, here: https://www.systemdesign.academy/interview/design-paypal
Top comments (0)