I thought the hardest part of building a reliable payment service was making sure the same payment never ran twice.
For one of my microservices projects, I built the payment flow around an internal wallet. It had the pieces I cared about at the time: idempotency, pessimistic locking for concurrent payments, transactional wallet and ledger updates, asynchronous communication, and basic failure handling. Within a system I controlled end to end, the design felt solid.
Then I started looking at what changes when the money movement leaves that boundary.
One failure scenario completely changed how I looked at the architecture: the payment provider successfully charges the customer, but my service crashes before recording the success. My database still says the payment is unfinished, while the provider has already moved the money.
That is where I realized duplicate prevention was only one part of payment reliability. A payment can continue beyond a single request, finish asynchronously, and leave two systems with different versions of what happened.
So I went back to the design with a different question: if I had to build this payment service again for failures that cross system boundaries, what would I change?
This is that redesign.
The Payment Service I Started With
Before introducing any external payment provider, I built the service around an internal wallet. When an order reaches the payment service, it checks the customer's balance, deducts the amount safely, records the transaction, and publishes the result so the rest of the order flow can continue.
The flow looks roughly like this:
Each part of this design exists for a specific failure mode.
The payment request arrives asynchronously through Kafka, so duplicate delivery is something the service has to expect rather than treat as an exceptional case. Before executing the payment, the service checks whether that logical operation has already been processed.
For concurrent payments, the wallet is read using a pessimistic write lock. If two orders for the same customer arrive almost together, only one transaction can modify the wallet balance at a time. The second payment sees the balance left by the first instead of making a decision from stale data.
The actual debit and ledger entry then happen inside the same database transaction. If writing the ledger fails after the balance has been modified, the entire transaction rolls back instead of leaving the wallet and transaction history out of sync.
A successful movement leaves an auditable record such as:
Customer: C101
Order: O123
Type: DEBIT
Amount: ₹500
Status: SUCCESS
Once the transaction commits, the payment result is published back into the asynchronous order flow.
For the system I was experimenting with, these guarantees covered the important risks: duplicate processing, concurrent balance updates, partial database writes, and reliable communication between services.
But this architecture has one very useful property that is easy to overlook: the money movement still happens entirely inside infrastructure I control. The wallet, ledger, locking, and transaction boundary all belong to the same system.
The moment an external payment provider enters that flow, that assumption disappears.
Where the First Assumption Breaks
The original design works because the important financial state stays inside one system. The wallet balance, ledger entry, and payment result can all be coordinated within the same database transaction.
I intentionally kept the first version that way. The goal was to learn microservices and distributed-system behaviour, not to integrate a real payment provider.
The failure model changes the moment the payment crosses that boundary.
Imagine my payment service sends a ₹500 charge to an external provider. The provider completes the payment successfully, but my service crashes before saving the result.
Now both systems hold different versions of the same payment:
Payment provider: SUCCEEDED
My payment service: PROCESSING
From my database, the payment still looks unfinished. From the provider's side, the money has already moved.
This is where the guarantees from the first design stop helping. A local transaction can roll back changes inside my database, but it cannot undo an operation that has already completed in another system. The pessimistic wallet lock does not help either; there is no longer a shared row to lock. The uncertainty now exists across a network boundary.
The obvious reaction is to retry the payment.
But before doing that, I need to answer a much more important question:
Did the first attempt actually fail, or did I simply lose the response?
That question becomes the starting point for the redesign. Every external payment operation needs an identity that remains stable even when the request is retried, the connection fails, or my service restarts.
The First Fix: Give Every External Payment a Stable Identity
Once the payment crosses into a third-party provider, retries become unavoidable. A timeout does not tell me whether the provider never received the request, started processing it, or completed the payment and lost the response on the way back.
So the first thing I would add to the redesigned service is a stable idempotency key for every external payment operation.
For example:
Payment: PAY-1042
Amount: ₹500
Provider: Stripe
Idempotency-Key: pay_1042_attempt_1
If the connection drops, I should not generate a new key and create another operation. I retry the same request with the same key.
Stripe's behaviour is a useful reference here. Once endpoint execution begins, Stripe associates the idempotency key with the result of that request. A retry with the same key and the same parameters returns the previous status code and response body instead of executing the operation again. If the same key is reused with different parameters, Stripe rejects it.
Even a 500 is interesting here. Once execution has started, a server error does not necessarily mean that no side effect occurred. Replaying the stored result is safer than blindly assuming the payment never happened and executing it again.
The recovery path now looks much safer:
Request 1
PAY-1042 + ₹500 + K1
↓
Provider processes payment
↓
Response is lost
Request 2
PAY-1042 + ₹500 + K1
↓
Provider recognizes the operation
↓
Previous result is returned
This protects a different boundary from the idempotency I already had inside my own system. Local duplicate protection stops the same payment command from being processed twice by my services. Provider-side idempotency protects the call between my payment service and the system actually moving the money.
I need both because duplication can happen on either side of that boundary.
There is also an important limit to what an idempotency key can do. Stripe can remove idempotency records after they have been retained for at least 24 hours. If that key is later reused after it has been pruned, it can be treated as a new request.
That means an idempotency key is useful for making retries safe, but it cannot become the permanent identity of the payment.
A payment can live much longer than a retry window. The customer might switch payment methods, complete authentication later, or remain in a processing state while the provider finishes the transaction.
So the redesigned service needs something more durable than a request key.
A Payment Needs a Lifecycle, Not Just a Request
The next gap appears when the payment itself lasts longer than the API call that started it.
My original flow is simple: receive the payment request, validate it, process it, and return success or failure. That works well for an internal wallet because the operation is usually immediate and fully controlled by the payment service.
External payments are not always that clean. A card can be declined and replaced with another payment method. The customer may need to complete 3D Secure authentication. Some payment methods can remain in a processing state before the provider knows the final result.
At that point, I need to separate the payment itself from the individual attempts used to complete it.
Stripe's PaymentIntent model gave me a useful way to think about this. Instead of treating every attempt as a new payment, one persistent object represents the payment for an order while different attempts and state changes happen underneath it.
I would bring the same idea into the redesigned service with a durable Payment record:
Payment: PAY-1042
Order: O-501
Amount: ₹500
Status: PROCESSING
PAY-1042 remains the same even if the first card fails, the customer switches payment methods, authentication is required, or the provider takes time to finish processing.
Its state can move through stages such as REQUIRES_ACTION, PROCESSING, FAILED, or SUCCEEDED without creating a new payment every time something changes.
This creates an important separation in the design: the idempotency key identifies a particular operation or retry, while the Payment record represents the longer-lived financial process those operations belong to.
And once the payment can live beyond the original request, the service also needs a way to learn about changes that happen later.
Some Payments Finish After the Request Is Over
Once a payment has its own lifecycle, the next question is how my service learns about changes that happen after the original request has already finished.
A payment might remain in PROCESSING, wait for bank confirmation, require authentication, or complete a few seconds later. Keeping the original HTTP request open until all of that finishes is not a reliable option, so the provider needs another way to notify my system.
That is where webhooks fit in.
What stood out to me was how familiar this problem looked. A webhook behaves a lot like an event consumed from Kafka: delivery can fail, the same event can arrive more than once, and events may not arrive in the exact order they were created.
So I would treat the webhook handler as another idempotent consumer. When the provider reports that PAY-1042 has succeeded, the service first verifies the event, checks whether that event has already been processed, and only then applies the corresponding payment-state transition.
If the same event arrives again, it should become a harmless duplicate rather than trigger the order flow twice.
This gives the redesigned service a reliable asynchronous path for normal payment updates. But webhooks still do not guarantee that both systems will always stay in sync.
If my database continues to say PROCESSING while the provider already considers the payment successful, I need a way to determine which state reflects what actually happened.
That is where reconciliation becomes necessary.
When Retrying Is No Longer the Right Answer
Idempotency, a persistent payment record, and webhooks reduce a lot of uncertainty, but they still cannot guarantee that my system and the provider will always agree.
Suppose PAY-1042 has been sitting in my database as PROCESSING. My service may have crashed after the provider completed the payment, or a webhook may have failed before the local update was committed.
Meanwhile, the provider already considers the payment successful:
My payment service
PAY-1042 = PROCESSING
Payment provider
PAY-1042 = SUCCEEDED
At this point, retrying the charge is the wrong recovery strategy. The customer may already have paid.
This is where reconciliation becomes part of the design.
Reconciliation compares independently recorded versions of the same financial transaction, detects when they disagree, and brings the inconsistent state back in line with what actually happened.
For PAY-1042, the reconciliation process can query the provider using the stored provider payment reference, compare that state with my local record, and repair the mismatch:
Before reconciliation
Local → PROCESSING
Provider → SUCCEEDED
After reconciliation
Local → SUCCEEDED
Provider → SUCCEEDED
The important distinction for me was this: retries deal with uncertain execution; reconciliation deals with uncertain truth.
If I am still trying to safely execute an operation, retry logic helps. If the operation may already have happened and two systems disagree about the result, I need to establish the actual payment state before doing anything else.
That is why the redesigned service needs to retain durable payment and provider references long after the original API request is over.
PaymentIntent, Authorization, and Charge Are Different Things
One distinction I would keep explicit in the redesigned model is the difference between the payment lifecycle and the actual movement of money.
A PaymentIntent represents the payment I am trying to complete for an order. It can survive failed attempts, authentication steps, payment-method changes, and different status transitions while still representing the same logical payment.
A Charge is closer to a concrete attempt to move money. It records details such as the amount, payment method, outcome, captured amount, and refunded amount.
This becomes important when a payment is not settled immediately.
For some flows, the first step is authorization. The provider confirms that the payment method can cover the amount and places a temporary hold on those funds. The merchant can then capture the authorized amount later.
A hotel is a simple example. It might authorize ₹10,000 at check-in to reserve the funds, then capture the final bill when the stay ends. If the authorization is never captured and eventually expires, the hold is released.
In the redesigned service, I would therefore keep these concepts separate:
Payment
PAY-1042
→ the complete payment lifecycle
Authorization
→ reserves funds for a possible later capture
Charge
→ a concrete payment attempt and its financial result
Capture
→ completes an authorized payment
This separation also makes later operations easier to reason about. If a captured payment has to be returned, a refund should be recorded as another financial operation with its own history. It is not equivalent to rolling back the original transaction and pretending it never happened.
With these boundaries clear, the individual pieces of the redesign are in place. The next step is to connect them into one V2 architecture and see how the entire payment flow behaves under failure.
V2: Designing Around the Failure Boundary
By this point, the redesign is no longer about adding one more safety check to the original payment service. The boundary itself has changed.
In the first version, I controlled the wallet, the ledger, and the database transaction that connected them. In V2, part of the payment can happen in a system I do not control, complete after my request has ended, or succeed while my own database still says PROCESSING.
I still would not solve that by turning the payment service into a collection of new microservices. The design can remain fairly small. The payment service owns the lifecycle, a durable database stores the financial state, a provider adapter handles external calls and retry policy, webhooks bring asynchronous updates back into the system, and reconciliation handles payments whose state remains uncertain.
The diagram intentionally has fewer boxes than the number of failure cases we have discussed. Inbox and Outbox are persistence patterns inside the payment boundary, not independent services. Exponential backoff and jitter belong inside the provider adapter. Authorization, capture, retries, refunds, and individual attempts are behaviours around the payment lifecycle rather than separate architectural components.
Running one payment through V2
Suppose an order for ₹500 reaches the payment service through Kafka.
The first job is to establish which logical payment this command belongs to. If Kafka delivers the same command twice, the second delivery should resolve to the same operation rather than create another payment.
The service can create a durable record such as PAY-1042, linked to the order, amount, payment method, current state, and eventually the provider's payment reference.
That record is important because it outlives the message that created it.
When the service is ready to contact the provider, the provider adapter sends the request using a stable idempotency key. If the call times out, the retry uses the same key. It does not create another operation simply because the response was unclear.
Retrying also needs a policy.
If the failure looks transient, such as a connection timeout or temporary provider unavailability, the next attempt should not happen immediately in a tight loop. I would use exponential backoff so the delay grows after repeated failures, then add jitter so many clients recovering from the same incident do not retry at exactly the same moment.
The more important decision, though, happens before the retry.
A timeout, an invalid request, and a declined card are three different failures. Retrying all of them would waste resources and, in some cases, make the behaviour worse. Transient infrastructure failures may deserve another attempt. A permanent validation error or card decline should instead move the payment into the appropriate business state.
This is also why an ambiguous 500 deserves care. If the provider already started executing the request, an internal error does not automatically prove that nothing happened. With providers that support result replay for idempotent operations, the safer recovery path is to retry the same operation identity and recover the previous result rather than create a new payment attempt.
The key only solves that short-term execution problem. It is not where I would keep the long-term identity of the payment. The durable Payment record remains responsible for that, including the provider payment ID once one exists.
The payment can keep moving after my request returns
Now suppose the provider accepts the payment but returns PROCESSING.
That does not mean the operation failed. It means the final outcome is not known yet.
The same PAY-1042 can continue through the lifecycle while the customer completes authentication, the provider waits for confirmation, or an authorized payment waits to be captured.
The useful property of this model is that the payment identity remains stable while the execution underneath it changes.
A failed card attempt does not necessarily end PAY-1042. Another payment method can be attached and tried. A payment that requires authentication can wait for the customer to complete it. An authorized payment can remain in REQUIRES_CAPTURE until the later capture operation finishes it.
Each external mutation can still have its own operation identity, but all of them belong to the same payment lifecycle.
I would keep refunds outside this state machine. Once a successful payment has already moved money and later needs to be reversed, the refund becomes another financial operation. The ledger should preserve both events instead of rewriting history as though the original payment never happened.
That gives the payment model a useful separation: the lifecycle tells me what is happening to the payment, while the ledger tells me what financial movements actually occurred.
Webhooks close the asynchronous path
Once the original request is over, the provider still needs a way to tell my system that something changed.
That is the job of the webhook path.
Suppose the provider later reports that PAY-1042 succeeded. Before applying that transition, the webhook handler verifies the event and records its event ID in the Inbox.
If the same provider event arrives again, the existing Inbox entry tells the service that the event has already been handled. The second delivery becomes a harmless duplicate instead of running the business transition twice.
This is effectively the external version of a problem I already had with Kafka. Both boundaries can deliver something more than once, so both consumers have to tolerate repetition.
When a valid new event changes PAY-1042 from PROCESSING to SUCCEEDED, I would commit that state change together with any required ledger update and an Outbox record such as PaymentSucceeded.
That outbox record protects another small but dangerous window.
Without it, this sequence is possible:
Payment DB commit succeeds
Payment becomes SUCCEEDED
service crashes
Kafka event never gets published
The payment is correct locally, but the rest of the system never learns about it.
By writing the outbox event in the same local transaction as the payment update, the event survives the crash. A publisher or CDC process can send it to Kafka later.
So the local transaction still matters in V2. Its job has simply become more precise: it cannot make the provider and my database one transaction, but it can keep my own payment state, ledger, inbox/outbox records, and local events consistent with each other.
When the normal path still leaves uncertainty
There is one remaining case that none of these mechanisms completely removes.
Imagine PAY-1042 has remained PROCESSING longer than expected. No useful webhook has repaired it, but the provider now shows the payment as SUCCEEDED.
Local payment: PAY-1042 = PROCESSING
Provider payment: SUCCEEDED
This is where the reconciliation job earns its place in the architecture.
It does not need to run on every payment request. I would keep it off the hot path and use it to inspect payments that have remained uncertain beyond an expected window.
For each one, it can query the provider using the stored provider payment reference and feed the result back through the same payment lifecycle.
If the provider confirms that the ₹500 payment already succeeded, V2 repairs PAY-1042 and continues the order flow from there. It does not create another charge simply because the local record was stale.
This also gives me a cleaner recovery hierarchy.
Normal retries handle transient request failures. Webhooks carry the expected asynchronous state changes. Reconciliation handles the smaller set of cases where the local system and provider still drift apart.
The important part is that all three paths eventually converge on the same durable Payment record rather than inventing separate versions of the transaction.
By the end of the redesign, the architecture is still relatively small. The additional reliability mostly comes from making the boundaries explicit: one identity for the payment lifecycle, stable identities for external operations, disciplined retries at the provider boundary, idempotent event consumption, reliable event publication, and a recovery path for state disagreement.
That is the main difference from my first design. V1 worked because the critical financial operation stayed inside a boundary I controlled. V2 is designed for the moment that assumption is no longer true.
Conclusion
When I first built this payment service, I was focused on the failures inside a system I controlled: duplicate execution, concurrent balance updates, transaction consistency, and reliable event flow. For that boundary, the design made sense.
What changed my thinking was moving the payment outside that boundary. Once an external provider is involved, the difficult part is no longer just preventing the same request from running twice. A payment can succeed while my service crashes, remain in progress after the original request ends, or leave my database and the provider holding different versions of what happened.
The V2 architecture grew from those failure cases rather than from adding infrastructure for its own sake. Each new piece exists because a specific uncertainty needed a recovery path.
The biggest lesson I took from this redesign is simple: reliable payment systems are ultimately about preserving the correct financial state, even when requests fail, services restart, events repeat, and the truth is temporarily split across multiple systems.
That is how I would approach the payment service if I were designing it again today.
📖 Blog by Naresh B. A.
👨💻 Backend & AI Systems Engineer | Distributed Systems · Production ML
🌐 Portfolio: [Naresh B A]
📫 Let's connect on [LinkedIn] | GitHub: [Naresh B A]
Thanks for reading. This is my personal engineering perspective, and I'd genuinely be interested in hearing where you agree or disagree. ❤️
Top comments (0)