Payments are the hardest problem in backend engineering, not because the code is complicated, but because the failure modes are unforgiving. A social feed can drop a post and recover. A payment system that charges a customer twice, loses a successful charge, or records a refund that never happened has created a real financial discrepancy, and that discrepancy will be discovered by a customer, an accountant, or a bank, in that order of unpleasantness. Stripe processes payments at enormous scale across many currencies, payment methods, and regulatory regimes, and it does so without losing track of money. The techniques it relies on are not secret, and they are available to any team building payment features. They are worth studying because they generalize to any system where correctness matters more than speed.
The central idea is idempotency. In ordinary APIs, sending the same request twice usually creates two effects: two rows, two emails, two charges. In a payment API, that is unacceptable, because networks are unreliable and clients will retry. A client that sends a charge request and never receives a response cannot know whether the charge succeeded. The safe behavior is to retry, but retrying must not create a second charge. Stripe solves this with idempotency keys. The client generates a unique key for each logical operation and sends it with the request. The server stores the result of the first execution against that key, and if the same key arrives again, it returns the stored result instead of executing again. The retry becomes safe, which changes the entire failure calculus. Clients can retry aggressively on timeouts, because the server guarantees that a retry of the same operation will not duplicate its effect.
Idempotency keys only work if they are used correctly, and the details matter. The key must be unique per logical operation, not per HTTP attempt, and it must be generated by the client before the first attempt, so that retries carry the same key. A key that is generated fresh on every retry is useless. The server must also store the result durably and atomically with the operation itself, so that a crash between executing the charge and recording the result cannot leave the system in a state where the charge happened but the key record did not. Stripe's API makes this explicit by accepting an idempotency key header on mutating requests, and by scoping keys to an account, which prevents collisions between unrelated operations. The lesson for any payment system is to design idempotency in from the start. Retrofitting it after duplicate charges appear in production is painful, because the duplicates are already in the ledger.
The second idea is the ledger. Stripe does not treat a payment as a single row that gets updated in place. It treats money movement as a sequence of immutable entries that sum to a balance. A charge creates entries, a refund creates offsetting entries, a payout creates entries, and the current state of an account is derived from those entries rather than stored as a single mutable number. This double entry style approach has a powerful property: errors become visible. If entries do not sum correctly, reconciliation catches it. If a row is updated in place and the update is lost, nothing in the data reveals the loss. If an entry is missing from a ledger, the totals do not reconcile, and the discrepancy points directly at the gap. Financial systems have used ledgers for centuries for exactly this reason, and Stripe's architecture follows that tradition in database form.
State machines are the third idea, and they govern how a payment moves through its lifecycle. A payment intent in Stripe moves through states such as requires payment method, requires confirmation, processing, succeeded, and canceled, and only certain transitions are legal. The API enforces those transitions, so a client cannot refund a payment that never succeeded, or capture a payment that was already canceled. That enforcement matters at scale because it turns a large class of bugs into rejected requests instead of corrupt data. Developers building their own payment flows should model the same discipline: define the states, define the allowed transitions, enforce them in one place, and never let two different code paths update payment status with ad hoc rules. When status can only change through defined transitions, auditing becomes possible, because every change has a reason and a previous state.
Underneath all of this is the database problem. Payments require strong consistency for the operations that move money, and that means transactions, carefully scoped. A charge that debits one account and credits another must succeed or fail as a unit. A refund that reduces a balance and creates a refund record must do both or neither. Stripe's systems use transactional stores for these critical paths, and they accept the cost of that consistency, because the alternative, eventual consistency for money movement, creates windows where the system reports a state that is not true. That does not mean every part of a payment platform needs the strongest consistency level. Read models, dashboards, and analytics can be eventually consistent. The ledger itself, and the transitions that change it, cannot.
Retries and webhooks form the next layer. Payment processing involves third parties: banks, card networks, and wallets, each with their own latency and failure modes. A charge may be authorized but not yet captured, a bank may respond slowly, or a webhook delivery may fail. Stripe handles this by treating its own system as the source of truth and treating external events as inputs that must be reconciled, not as facts that can be trusted blindly. Webhook handlers must themselves be idempotent, because webhooks are delivered at least once and may arrive duplicated or out of order. A webhook handler that credits an account on every delivery will double credit on a duplicate delivery. A handler that checks the event identifier against already processed events, and ignores duplicates, will not. That pattern, store the event id, check before applying, apply transactionally, appears everywhere in payment systems, and skipping it is one of the most common causes of real money bugs.
Consider a simplified example. A service receives a webhook for a successful payment, and it must mark an order as paid exactly once, even if the webhook arrives three times. The correct structure is a table of processed event identifiers, checked inside the same transaction that updates the order. If the event was already processed, the handler returns success without changing anything. If it was not, the handler records the event and updates the order together, so a crash cannot leave one without the other.
# Webhook handling must be idempotent: the same event delivered
# twice must not apply its effect twice.
def handle_payment_succeeded(event_id, order_id, db):
with db.transaction():
already = db.query(
"SELECT 1 FROM processed_events WHERE event_id = %s",
(event_id,),
)
if already:
return "already processed"
db.execute(
"INSERT INTO processed_events (event_id) VALUES (%s)",
(event_id,),
)
db.execute(
"UPDATE orders SET status = 'paid' WHERE id = %s",
(order_id,),
)
return "ok"
Reconciliation is the safety net that catches what idempotency and transactions miss. Stripe reconciles its internal ledger against bank statements, card network reports, and payout records on a regular cycle. Any discrepancy becomes an investigation item, not a silent drift. Smaller teams should adopt a version of the same discipline: a daily job that compares the sum of charges, refunds, and payouts in the application database against the payment provider's reports, and alerts on any mismatch. Reconciliation is not glamorous, and it is tempting to skip it when volumes are small. That is exactly when it is cheapest to build, and exactly when it prevents the small discrepancies that become large ones as volume grows.
None of this requires Stripe's scale to be useful. A small application that sells a single product can implement idempotency keys, a ledger style record of transactions, a payment state machine, idempotent webhook handling, and daily reconciliation, and each of those pieces will prevent a real class of bug. The scale changes the volume, not the principles. Money must be accounted for exactly, retries must be safe, state transitions must be constrained, and external events must be verified before they change internal state. Systems that honor those principles can grow from hundreds of transactions to millions without redesigning their core. Systems that ignore them will eventually face a discrepancy that no amount of manual correction can fully explain, because the data that would explain it was never recorded in the first place. That is what it means, in practice, to handle payments without losing a cent: not that failures never happen, but that every failure leaves a trace, and every trace can be reconciled.
Top comments (0)