Every Distributed Payment System Lies: How We Prevent Money Loss

Mohammad Dadashi — Wed, 08 Jul 2026 14:25:18 +0000

"The customer was charged, but the order doesn't exist."
If you've worked on payment systems long enough, you've probably heard a sentence like this.
It usually starts with a timeout.
The payment gateway successfully charges the customer's card.
Right before our service commits the database transaction, Kubernetes kills the pod.
The mobile application never receives a response.
After three seconds, it retries the request.
Congratulations.
You just charged the customer twice.
This is why experienced backend engineers don't chase exactly-once processing. We build systems that behave as if operations happened exactly once—even when the network retries requests, brokers redeliver messages, or services crash halfway through a transaction.
This article covers the patterns we've used to build resilient financial systems that don't lose money when distributed systems inevitably fail.

The Myth of Exactly Once

One of the biggest misconceptions in distributed systems is that "exactly once" is something you can simply enable.

You can't.

Every network call can experience one of three outcomes:

The request never reaches the server.
The request succeeds.
The request succeeds, but the response never reaches the client.

The third scenario causes most production incidents.

Client
   │
   │ Payment Request
   ▼
Payment Service
   │
   │ Charge Card
   ▼
Payment Gateway
   │
   │ Success
   ▲
   │
Network Timeout

From the client's perspective, the request failed.

From the payment gateway's perspective, it succeeded.

Who is right?

Both.

That's why clients retry.

And retries are exactly where duplicate payments begin.

Retries Are Good. Non-Idempotent APIs Are Dangerous

Imagine this endpoint.

@PostMapping("/payments")
public PaymentResponse pay(@RequestBody PaymentRequest request) {
    return paymentService.process(request);
}

The client retries.

The controller executes again.

The payment gateway receives another charge request.

Nothing prevented it.

The bug wasn't the retry.

The bug was assuming the request would only arrive once.

Every API handling financial operations should assume the same request may arrive multiple times.

Idempotency Is Your First Line of Defense

The simplest solution is surprisingly effective.

Every payment request carries a unique Idempotency-Key.

POST /payments

Idempotency-Key:
4f6a6d4d-8d8f-4d1d-89cf-29d51d42d667

Before processing the payment, we check whether this key already exists.

                Exists?
Request ─────► Database
                 │
         ┌───────┴────────┐
         │                │
       Yes               No
         │                │
Return saved       Process payment
response            Save result

A simplified implementation looks like this:

@Transactional
public PaymentResponse process(String key, PaymentRequest request) {

    var existing = repository.findById(key);

    if (existing.isPresent()) {
        return existing.get().response();
    }

    PaymentResponse response =
            gateway.charge(request);

    repository.save(
        new PaymentRecord(key, response)
    );

    return response;
}

Now the client can retry five times.

The payment is still executed once.

The Real Problem Starts After the Payment

Many engineers stop after implementing idempotency.

That isn't enough.

Consider this sequence.

Charge customer
Update database
Publish PaymentCompleted event

What happens if the application crashes here?

Charge Card ✔

Save Database ✔

Publish Event ✖

Money has been collected.

No other service knows about it.

Inventory is never reserved.

Email is never sent.

Order status remains "Pending."

The database and the rest of the system now disagree.

This is called the dual-write problem, and nearly every distributed system encounters it.

The Transactional Outbox Pattern

Instead of writing directly to Kafka or RabbitMQ, we first write the event into the same database transaction.

+----------------------+
| Payments Table       |
+----------------------+

+----------------------+
| Outbox Table         |
+----------------------+

       Same Transaction

@Transactional
public void completePayment(...) {

    paymentRepository.markCompleted(id);

    outboxRepository.save(
        new PaymentCompletedEvent(...)
    );

}

Later, a background publisher reads the Outbox table and publishes events.

Database

Payments
Outbox
   │
   ▼

Publisher

   │

Kafka

   │

Consumers

Now the database and the event are committed together.

If the service crashes, the event is still safely waiting in the Outbox table.

Nothing gets lost.

But Wait... What If Kafka Receives the Same Event Twice?

It absolutely can.

Imagine this timeline.

Publisher
     │
     │ Send Event
     ▼
Kafka

Publisher crashes

Database still says:
Published = false

When the publisher restarts, it sends the event again.

This is expected.

Outbox solves lost events.

It does not eliminate duplicates.

That responsibility belongs to the consumer.

Consumers Must Be Idempotent Too

Every event should contain a unique identifier.

Before processing, consumers check whether that identifier has already been handled.

Redis makes this extremely fast.

Boolean accepted =
    redis.opsForValue()
         .setIfAbsent(
             messageId,
             "DONE",
             Duration.ofHours(24)
         );

if (Boolean.FALSE.equals(accepted)) {
    return;
}

If Redis already contains the key, the event is ignored.

The message broker can deliver the same event multiple times without producing duplicate business operations.

What About Distributed Transactions?

Imagine a payment flow involving four services.

Payment

↓

Fraud Check

↓

Inventory

↓

Shipping

Suppose Shipping fails.

Do we keep the payment?

Do we release inventory?

Do we refund?

Traditional distributed transactions (2PC) attempt to solve this problem by locking every participant.

They also introduce blocking, coordinator failures, and poor scalability.

Modern fintech systems generally prefer Saga.

Instead of one giant transaction, every service performs its own local transaction.

If something fails later, compensating actions reverse the completed work.

Charge Payment ✔

Reserve Inventory ✔

Shipping ✖

↓

Refund Payment

↓

Release Inventory

The system is temporarily inconsistent, but eventually reaches a valid state.

That's a much better trade-off than globally locking every database.

Observability Is Part of the Design

Resilience isn't only about preventing failures.

It's about understanding failures quickly.

Every payment request should carry a correlation ID across every service.

Every metric should answer questions like:

How many retries happened today?
How many duplicate requests were ignored?
How many Outbox events are waiting?
What's the Kafka consumer lag?
Which payment failed, and where?

A production payment service without tracing is like trying to investigate a bank robbery with no cameras.

OpenTelemetry, Prometheus, Grafana, and structured logging are not optional additions.

They're part of the architecture.

Production Checklist

Before calling a payment service "production ready," I expect to see most of these:

Every write endpoint is idempotent.
Retry policies use exponential backoff.
Database writes and events use the Transactional Outbox pattern.
Consumers implement deduplication.
Dead Letter Queues exist for unrecoverable failures.
Correlation IDs flow through every service.
Metrics expose retries, failures, queue lag, and latency.
Distributed tracing is enabled.
Chaos testing verifies retry behavior.
Integration tests simulate broker failures and duplicate deliveries.

If even one item is missing, failures are usually a matter of when, not if.

Final Thoughts

One of the biggest lessons I learned while building distributed systems is that reliability doesn't come from perfect infrastructure.

It comes from assuming that infrastructure will eventually fail.

Networks partition.

Pods restart.

Message brokers retry.

Databases become temporarily unavailable.

Those aren't exceptional situations—they're normal operating conditions.

The goal isn't to eliminate failures.

The goal is to design systems where failures don't cost customers money.

Exactly-once processing remains a useful concept, but in real production systems we achieve something more practical: at-least-once delivery combined with idempotent business logic.

When those pieces work together—idempotency, retries, the Transactional Outbox pattern, consumer deduplication, Sagas, and strong observability—the result is a payment platform that behaves correctly even when everything around it is unreliable.

And that's ultimately what users care about.

They don't care how many times your service retried.

They only care that they were charged exactly once.

DEV Community: Mohammad Dadashi