A customer taps Pay. The screen spins, the request times out, and the customer taps again.
Did the first payment fail? Did the provider charge the customer but lose the response? Can the second tap safely retry? And if the payment succeeded, what happens when the order service never receives the confirmation?
These are the questions that make payment system design fundamentally different from a typical CRUD service. The difficult part is not sending a request to a payment gateway. It is preserving financial correctness when different systems observe different outcomes.
This article designs a one-time payment orchestration system, starting with a simple flow and evolving it through the failures that justify each architectural component. It covers idempotency, payment attempts, webhooks, transactional outbox, refunds, financial ledgers, reconciliation, provider outages, and scaling.
For an interactive walkthrough, see Payment System — Complete Design on SeeItFlow and the Visual Design Journey.
1. Define the payment system we are designing
We are building a service that accepts a customer's payment intent, coordinates with an external payment service provider (PSP), records the result, and reliably informs downstream systems.
It must support:
- Creating and retrieving a logical payment.
- Preventing accidental duplicate payments from client retries and concurrent requests.
- Handling immediate, pending, and ambiguous provider outcomes.
- Receiving authenticated provider webhooks.
- Reconciling local records with provider records.
- Issuing full or partial refunds when supported.
- Recording an auditable history of financial operations.
- Triggering downstream actions without coupling their availability to the payment request.
The baseline assumes one-time payments through an external provider. Subscription billing, card-network authorization internals, tax, FX, fraud scoring, and merchant settlement infrastructure are separate systems or extensions. Actual capabilities depend on the provider, payment method, country, and merchant configuration.
Core invariant: retries of one logical payment must not create additional independent charges. This is a design objective enforced by several layers, not an unconditional guarantee that an external financial system can never produce a duplicate.
2. Start with the simplest payment flow
Customer
|
v
Payment API --> Payment Database
|
v
Payment Provider (PSP)
|
+--> synchronous result or pending/next action
|
+--> later webhook / status lookup
A safe basic flow is:
- Authenticate and validate the request, including the merchant/order, amount, and currency.
- Atomically create or find the existing logical payment associated with its idempotency key.
- Persist a payment attempt and its stable provider-facing identity before contacting the provider.
- Send the operation through the provider adapter, using the provider's documented idempotency mechanism when available.
- Persist the outcome actually known: succeeded, definitively failed, requires customer action, pending, or unresolved.
- Return the current state and a stable payment identifier. Resolve nonterminal outcomes through subsequent provider messages and reconciliation.
The provider never writes directly to our database. Our service processes the provider's API response or webhook and performs its own guarded database updates.
The payment API should not tell the customer “paid” merely because the provider accepted an asynchronous request. Accepted, authorized, captured, and settled can represent different events.
3. Model payment intent separately from payment attempts
A single payment is a business operation: for example, paying ₹1,500 for order order_847.
An attempt is a particular interaction with a provider to achieve that operation.
Payment: pay_847 (INR 150000 minor units)
|
+-- Attempt A: Provider X, definitive decline
|
+-- Attempt B: Provider X, successful charge
The second attempt is permitted only after the system knows the first cannot also result in a charge, or after an explicitly designed recovery process. A timeout is not enough evidence to start another independent attempt.
A conceptual schema:
payments (
payment_id, merchant_id, order_id,
amount_minor, currency, current_state,
state_version, created_at
)
idempotency_records (
merchant_id, operation, idempotency_key,
request_fingerprint, payment_id,
UNIQUE (merchant_id, operation, idempotency_key)
)
payment_attempts (
attempt_id, payment_id, provider,
provider_idempotency_key, provider_reference,
attempt_state, created_at
)
The exact constraints depend on the business domain. If one order may legitimately have multiple payments, do not introduce a simplistic globally unique order_id constraint. Conversely, where the business requires only one successful capture for an order, enforce that invariant explicitly at the appropriate transactional boundary.
Use integer minor units or fixed-precision decimal for money, with an explicit currency. Do not use binary floating point for authoritative monetary calculations, and do not assume all currencies have two fractional digits.
4. Payment state is a state machine, not paid = true
A simplified flow:
INITIATED --> PROCESSING --> SUCCEEDED
| \
| --> REFUND operation (separate lifecycle)
+--> FAILED (definitive)
+--> UNRESOLVED (outcome unknown)
+--> REQUIRES_ACTION (where supported)
These are internal illustrative states, not a universal provider vocabulary. A real design may also distinguish authorization, capture, cancellation/void, expiry, partial capture, reversal, and partial refund. Some of these belong to separate operation-level state machines rather than a single overloaded payment status.
State changes must be conditional. For example:
UPDATE payments
SET current_state = :next_state,
state_version = state_version + 1
WHERE payment_id = :payment_id
AND current_state = :expected_state
AND state_version = :expected_version;
A zero-row update means the caller must reload and evaluate the current state; it does not automatically mean the provider event is wrong. Transition rules, provider event identity, and, where available, provider sequence/version information help avoid applying stale events.
A delayed PROCESSING webhook must not regress a confirmed successful capture. But refunds, reversals, and chargebacks are new financial facts, not simply stale messages to discard because an earlier payment succeeded.
5. Idempotency: protect the customer from duplicate requests
Suppose the client submits the same payment twice:
POST /payments
Idempotency-Key: checkout-847-v1
{"order_id":"order_847","amount_minor":150000,"currency":"INR"}
The first request creates a durable payment. The second, using the same key and identical meaningful parameters, resolves to that same payment rather than creating a new independent charge.
The implementation needs four pieces.
A durable uniqueness constraint. Scope the key to the appropriate merchant/account and operation. An atomic insert or equivalent transaction arbitrates concurrent requests. A Redis lookup or distributed lock alone is not sufficient: cache eviction, lock expiry, and crashes can erase the only deduplication signal.
A request fingerprint. The same key with a different amount, currency, order, or operation is a conflict, not a request to charge a new amount. Reject it explicitly.
In-progress behavior. If the original request is still running, return its current state, a pending response, or wait for a bounded period. Do not let the duplicate independently call the provider.
A separate provider-side idempotency boundary. Retrying the same provider attempt should reuse the same provider operation identity if that provider documents safe idempotent retries. Local uniqueness prevents duplicate logical records; provider idempotency helps prevent duplicate external effects. Neither mechanism substitutes for the other.
Provider idempotency retention periods and exact replay semantics differ. Do not assume a provider remembers a key forever. Keep durable local business identity and reconciliation records for the periods required by the product and applicable obligations.
Also, an idempotency key cannot deduplicate two logically identical requests if a buggy client generates a new key on every retry. Where required, supplement request-level keys with a stable order/payment-intent invariant.
6. The most important failure: provider timeout
Consider this sequence:
Payment Service Provider
| |
|--- charge attempt ---->|
| | charge succeeds
|<--- response lost ------X
|
| local outcome: UNKNOWN
The timeout says only that our service did not receive a conclusive response. The request might never have arrived, might still be processing, might have failed, or might already have succeeded.
Never equate a timeout with a declined payment. And never automatically create a new attempt with a new provider identity because the old attempt timed out.
A safe resolution strategy depends on the provider contract:
- Query the provider using the stable attempt/reference ID, where supported.
- Retry the same provider operation with the same idempotency identity only if the provider documents that as safe.
- Accept an authenticated webhook reporting the outcome.
- Escalate unresolved attempts to operational reconciliation and, when necessary, financial reconciliation.
These channels can operate in parallel or in a different order; the provider's documented semantics determine which is safest. A provider API status response may itself be provisional, so do not treat every response as a final settlement fact.
If the process crashes after the provider succeeds but before local SUCCEEDED is committed, the persisted attempt and provider reference make later recovery possible. If the local success commit happens but the API response is lost, a retry with the same client key returns the existing payment.
This is why durable identity must exist before the external side effect.
7. Webhooks are necessary, but they are not a single source of guaranteed truth
Many payment methods finish asynchronously. The provider notifies us later:
Provider --> Webhook Endpoint --> Durable Inbox --> Worker
|
v
Payment Database
A robust webhook handler:
- Verifies the provider's documented authentication/signature mechanism against the exact bytes and headers required by that mechanism; checks timestamp/replay protection where supported.
- Durably records the incoming event or equivalent recoverable work item.
- Acknowledges according to the provider's retry contract.
- Processes it asynchronously and idempotently.
- Applies a guarded state transition and records any consequential event or ledger posting with its own deduplication key.
Do not assume all providers sign the same payload or use the same event-ID semantics. A delivery ID is not always the same as a business-event ID; deduplication should use the provider's documented stable identifier, plus the underlying operation identity where needed.
Expect duplicate delivery, out-of-order events, missing webhooks, and messages arriving after a status query has already resolved the payment. A signature-valid webhook also does not eliminate the need to check that the event belongs to the expected merchant, payment, amount, currency, and operation.
A webhook for an unknown provider transaction should be investigated or quarantined rather than blindly creating a successful payment record.
8. The database-to-queue dual-write problem
Once a payment is confirmed, the system may need to release an order, grant access, notify the customer, and update financial records.
A naïve implementation does this:
1. UPDATE payments SET state = 'SUCCEEDED'
2. PUBLISH payment.succeeded
What if the database commit succeeds and the process crashes before publishing? The payment is recorded, but the order service never hears about it.
The transactional outbox solves that gap:
One local database transaction:
- apply guarded payment state transition
- insert outbox event with stable event_id
Commit
Outbox relay --> Message Broker --> Consumers
The relay may publish the same event twice if it crashes after broker acknowledgement but before recording publication. Therefore the outbox supports reliable at-least-once delivery, not magical end-to-end exactly-once effects.
Each consumer needs its own idempotent business operation. For example, order fulfillment can enforce uniqueness on the payment/fulfillment operation it has already applied. Deduplicating by event ID alone may be insufficient if different event IDs represent the same underlying business action.
An important accounting design choice: if the authoritative ledger and payment state live in the same transactional database, write both in the same local transaction when appropriate. If ledger posting is asynchronous, explicitly monitor and reconcile the gap between payment confirmation and ledger posting; do not imply the outbox itself makes the ledger instantly consistent.
9. Payment state, ledger, and settlement are different
These concepts are often conflated in system design interviews.
| Record | Question it answers | Typical property |
|---|---|---|
| Payment state | What is the workflow's current status? | Guardedly mutable |
| Audit history | What changed, when, and by which actor/system? | Append-only or tamper-evident |
| Financial ledger | Which monetary obligations or movements have been recorded? | Append-only journal with corrections as new entries |
| Provider settlement report | What did the provider actually settle, deduct, or transfer? | External reconciliation evidence |
A financial ledger commonly uses double-entry bookkeeping: each journal transaction contains balanced postings across accounts under the chosen accounting model. A ledger entry is not automatically proof that funds have settled in the bank account.
For example, an authorized card payment is not the same as a captured payment; a captured payment is not necessarily already settled to the merchant. Provider fees, reserves, disputes, reversals, and payout timing may affect reconciliation.
Avoid saying that a payment.succeeded event means cash is already in the merchant's bank account. What it means depends on the provider event and payment method.
10. Authorization, capture, void, and refund
Not every payment should be charged immediately.
Immediate charge/capture can suit purchases where fulfillment and amount are known upfront.
Authorize then capture can suit hotels, rentals, or orders where inventory or final amount needs confirmation. An authorization generally reserves or approves funds subject to network/provider rules; capture is a later step requesting the charge. Authorizations can expire, and partial/multiple captures are provider- and payment-method-dependent.
Void/cancel typically applies before capture or while an operation remains cancellable. Refund is a separate operation after a charge/capture, subject to provider rules and payment state.
Refunds need their own IDs, attempts, idempotency identities, statuses, and reconciliation. A refund timeout is ambiguous for the same reason a charge timeout is ambiguous.
For partial refunds, protect the available refundable amount against concurrent requests. A safe local reservation/transaction must account for successful and still-in-flight refund amounts, not merely the sum of completed refunds, otherwise two concurrent provider calls can each appear individually valid and together exceed the captured amount. Provider-side refund limits remain an additional protection, not a substitute for local coordination.
Refund ledger postings should be new entries, not edits that erase the original charge history.
11. Reconciliation is a core subsystem, not an emergency script
Webhooks are not guaranteed to arrive promptly or exactly once. Provider APIs and internal databases can temporarily disagree. Reconciliation detects and resolves the difference.
Operational reconciliation checks in-flight, stale, and unresolved operations against provider APIs or other supported status channels. It can identify a charge that succeeded externally but never updated the internal state.
Financial reconciliation compares internal financial records against provider reports, settlement exports, fees, payouts, and adjustments. It can reveal mismatched amounts, missing transactions, unexpected duplicates, refunds, and reversals that payment-status polling alone would not catch.
A production reconciliation job should have:
- Stable provider and internal reference mappings.
- Bounded, restartable batches with checkpoints or watermarks.
- Rate-limit-aware status lookups.
- Idempotent repair operations with durable action IDs.
- Exception queues for unresolved mismatches requiring investigation.
- Alerts for aging unresolved payments, ledger gaps, and settlement differences.
Do not automatically trust whichever record has the newest timestamp. Interpret the provider's event semantics and the settlement evidence, then record corrections without silently rewriting financial history.
12. Multi-provider routing has a double-charge trap
A second provider can improve availability, but failover is not simply retrying elsewhere.
Provider A: request timed out, but charge actually succeeded
Provider B: automatic failover charge also succeeds
Result: customer may be charged twice
Different providers generally cannot deduplicate each other's charges using a shared idempotency key.
Only route a new independent attempt when the previous attempt is known not to have created a charge, or when a specifically designed orchestration and recovery protocol makes that decision safe. Keep each provider attempt separately identifiable under the same logical payment.
Circuit breakers, rate limits, exponential backoff, and jitter protect capacity, but none of them resolves an already ambiguous financial outcome. A 5xx is not universally proof that a provider performed no side effect; follow that provider's error semantics.
13. Security and compliance boundaries
For most integrations, prefer provider-hosted payment collection or tokenization so your application avoids directly handling raw card details wherever possible.
Tokenization can reduce PCI DSS scope, but does not automatically remove all compliance obligations. Exact scope depends on how the integration works, what systems can affect the payment page, and the applicable standards and assessments.
Keep payment API authorization strict: a user must not retrieve or refund another merchant's payment by guessing an identifier. Verify webhook authenticity, prevent replay where the provider supports it, protect credentials, limit sensitive data in logs, and audit privileged refund and payment-data access.
Never store card verification values after authorization; any handling of cardholder data must follow the applicable payment security rules. Payment references, personal information, and webhook payloads can also be sensitive even when they are not raw card numbers.
14. Scaling without weakening correctness
Payment APIs can be horizontally scaled because authoritative state belongs in durable systems, not process memory. The harder bottlenecks tend to be database contention, provider quotas, webhook throughput, and reconciliation backlog.
A pragmatic progression:
- Start with a strongly consistent transactional store for payment identity, idempotency, attempts, and guarded state changes.
- Add appropriate indexes for merchant/payment lookup, unresolved attempts, provider references, and reconciliation scheduling.
- Scale webhook processing, outbox relay, and downstream workers independently with bounded concurrency and backpressure.
- Use replicas for reporting and stale-tolerant reads, not for authoritative decisions such as whether a charge already exists.
- Partition or shard only when measured bottlenecks justify the complexity; preserve uniqueness and ownership invariants across partitions.
Multi-region payment writes require especially careful ownership and fencing. If two regions can simultaneously accept the same idempotency key without a shared uniqueness guarantee, each may initiate a separate provider charge. A failover design must also preserve or recover in-flight attempt identities, not just database rows.
Operational metrics should include provider timeouts by operation, ambiguous-attempt age, idempotency conflicts, webhook verification failures and processing lag, outbox age, refund reservation backlog, reconciliation mismatches, and ledger/settlement discrepancies. Ordinary request latency alone cannot tell you whether a payment system is financially correct.
15. Walk through the failures that matter
| Failure | Unsafe reaction | Safer design |
|---|---|---|
| Client retries after timeout | Create a new charge | Reuse logical payment and provider attempt identity |
| Two concurrent payment requests | Both pass a cache check | Atomic durable idempotency constraint |
| Provider response is lost | Mark payment failed | Preserve unresolved state; query, safely retry, or reconcile |
| Webhook delivered twice | Reapply financial effects | Durable event/business-operation deduplication |
| Older webhook arrives late | Overwrite latest state | Guarded transitions and provider semantics |
| DB commits but event publish fails | Lose downstream confirmation | Transactional outbox |
| Refund requests race | Each sees old refundable total | Transactionally reserve refundable amount |
| Provider A times out | Immediately charge at Provider B | Resolve A before independent failover |
| Payment recorded but ledger posting missing | Assume reporting will fix itself | Durable delivery, idempotent posting, monitored reconciliation |
| Primary region fails | Accept writes everywhere | Explicit write ownership, fencing, recovery of ambiguous attempts |
For a fuller production failure analysis, see Payment System — Failure & Scale Scenarios.
16. The final architecture
Customer / Checkout
|
v
Payment API
|
v
Payment Orchestrator
/ | \
/ | \
Payment DB Provider Status API
+ Attempts Adapter |
+ Idempotency | |
+ Outbox v |
+ Ledger* External PSP <----+
| |
| v
| Webhook Endpoint
| |
| Verified Durable Inbox
| |
| Idempotent Worker
| |
+<---------+
|
Outbox Relay ---> Broker ---> Order / Notification
|
Reconciliation Jobs <---- Provider API / Reports
* Ledger may be in the same transactional boundary or a
separately operated financial subsystem, with explicit
consistency and reconciliation guarantees.
Every component exists for a reason:
- Payment orchestrator: one logical payment and guarded state transitions.
- Durable idempotency: duplicate client requests cannot create independent logical payments within the key's contract.
- Provider adapter: normalizes provider-specific operations and outcome semantics.
- Webhook inbox: authenticated, durable, duplicate-tolerant asynchronous updates.
- Outbox: durable handoff to downstream business processes.
- Ledger: auditable financial postings, distinct from workflow status.
- Reconciliation: finds and resolves external/internal discrepancies.
The architecture is not meant to imply that all components must be separate microservices. Several can begin as modules within one well-structured service and database.
17. How to explain this in a system design interview
A strong payment-system answer can follow this order:
- Clarify the payment method, synchronous versus asynchronous outcomes, authorization/capture, refunds, and provider count.
- Define logical payment identity, attempt identity, money representation, and the state machine.
- Walk through the happy path, persisting the attempt before the provider call.
- Demonstrate duplicate-request handling using durable idempotency.
- Introduce timeout ambiguity, safe provider retries, webhooks, and reconciliation.
- Close the database-to-queue gap with an outbox and idempotent consumers.
- Separate operational payment state from financial ledger and settlement.
- Discuss refunds, multi-provider failover, security, and scaling only to the depth the interview requires.
The strongest answers do not promise “exactly-once payments” without qualifications. They explain which boundary enforces which invariant, where external uncertainty remains, and how discrepancies are detected and corrected.
Explore the deeper decisions in Architecture Deep Dive, compare options in Trade-offs & Decisions, and practice the follow-up questions in Interview Perspective.
Final takeaway
A payment system is not correct merely because its happy path succeeds. It is correct to the extent that it can preserve durable identity, avoid duplicate external effects, represent uncertainty honestly, recover from crashes, and reconcile what it believes happened against what the provider and financial records show.
The question is not only “Did the API succeed?” It is “What financial effect actually occurred, how do we know, and what happens when we do not know yet?”
For the complete visual and technical learning path, explore Payment System Design on SeeItFlow.
Top comments (2)
The line I'd underline is "different systems observe different outcomes." I worked for years on a prepaid top-up platform where a request went through two, three, up to four hops before reaching the carrier, each with its own protocol and timeout. We saw both failure modes: double top-ups, and charges with no top-up. What made the difference was treating the timeout as a third state, unknown, instead of a failure, plus an ID created at the terminal that travels through every hop, so a retry gets the first result back instead of becoming a new top-up. With the carrier, the real answer often only came the next day, in reconciliation. In your design, when the PSP has no status endpoint, what decides how long a payment stays pending before it is marked failed?
Separating the payment intent from its attempts is the detail most teams skip and it shows up later as bugs nobody can reproduce, because a retry that created a second provider-side charge is indistinguishable from a duplicate when both live under one order id. Persisting the stable provider-facing reference before contacting the provider deserves the same emphasis, since it is what makes an ambiguous timeout answerable by query instead of guesswork. Treating webhooks as resolution rather than source of truth also matches how providers actually behave: delivery is best-effort, so the state machine has to be drivable entirely from poll-and-reconcile if the hook stream goes quiet.