Why Payments Break When the Network Retries (and How I Stopped It)

Sidharth Yadav — Tue, 13 Oct 2026 12:09:29 +0000

A customer clicks “Pay”. The screen freezes, so they click again. Or the app sends the request, the network drops before the answer arrives, and the app sends it again.

Did the customer just pay twice?

In a financial system, that question matters. If the same payment gets recorded twice, the ledger is wrong. A customer paid $100, but the books say $200. Someone has to find out why, and fix it by hand.

I ran into this while building remitlytics-core, a backend I wrote with Java 21, Spring Boot 3 and PostgreSQL. It keeps a double-entry ledger, which means every transaction has a debit and a credit that must match. Here’s how I handle retries, and what testing showed me about my own code.

What idempotency means

An operation is idempotent if doing it twice gives the same result as doing it once.

Think of the call button on a lift. Press it once or six times, and one lift still arrives. A payment API should behave the same way: send the same payment twice, and it’s recorded once.

The idempotency key

Every ledger transaction in my project carries a key. The key is built from what happened and which invoice it was for:

String idempotencyKey = "PAYMENT_SETTLEMENT_" + invoice.getId();

The same invoice and the same event always produce the same key. A retry can’t sneak in under a new one.

Two layers of protection

The first layer is a check in the code. Before recording anything, the service asks whether this key already exists. If it does, it logs a warning and returns:

if (transactionRepository.existsByIdempotencyKey(idempotencyKey)) {
    log.warn("Ledger transaction already processed ...");
    return;
}

The second layer is the database. The ledger_transactions table has a unique constraint on the key (simplified here):

CREATE TABLE ledger_transactions (
    id UUID PRIMARY KEY,
    idempotency_key VARCHAR(255) NOT NULL UNIQUE,
    description VARCHAR(255)
);

The debit and credit entries are saved in the same database transaction as the parent row. So if the parent insert is rejected, the entries never get saved either.

Why the first layer isn’t enough

The code check has a gap. If two requests arrive at the same moment, both can ask “does this key exist?” before either has saved anything. Both get “no”, and both go ahead.

That’s why the constraint is the real protection. Only one insert can win, and Postgres rejects the other. Code can have timing gaps, but the database can’t be talked into storing the same key twice.

Testing it

I wanted to check this properly, so I wrote two tests against a real PostgreSQL 16 instance with my Flyway migrations applied. I didn’t use an in-memory database, because concurrent inserts behave differently there.

Test 1: the same payment, recorded twice, one after the other. The second call hit the code check and returned early. Result: 1 transaction, 2 entries.

Test 2: the same payment, recorded by two threads at once. Both threads passed the code check. Both logged that they recorded the transaction. Then Postgres rejected one of them:

ERROR: duplicate key value violates unique constraint
"ledger_transactions_idempotency_key_key"

The final data was still correct: 1 transaction, 2 entries. The database did its job.

What the test caught

My code has a catch block for exactly this error, meant to log it and treat the request as “already processed”. It never ran.

Hibernate delays the insert until the transaction commits, which happens after my try/catch has finished. So the exception escaped to the caller, and my “Successfully recorded” log line was printed before the insert had really happened.

The data was safe, but the behaviour wasn’t what I’d written the code to do. I would never have noticed without a concurrent test.

The lesson

Don’t rely only on application code to catch duplicates. Check in code for the fast, common case, but let the database enforce the rule. It’s the only part of the system that sees every write.

I haven’t fixed this yet. The options I’m looking at are catching the exception in the caller, outside the transactional method, or using INSERT ... ON CONFLICT DO NOTHING and checking whether a row was actually inserted. I’ll write a follow-up once I’ve tested one.

The code is here:

Sidharth-Async / remitlytics-core

Remitlytics isn't just another CRUD API. It is a strictly structured financial engine designed to handle invoice lifecycles, guarantee transaction integrity via a double-entry ledger, and communicate with the outside world through a fully decoupled, async webhook pipeline.

🚀 Remitlytics — Core Engine & Dashboard

An event-driven, fault-tolerant financial backend and dashboard built for resilience and data integrity.

Remitlytics isn't just another CRUD API. It's a strictly structured financial engine designed to handle multi-tenant invoice lifecycles, guarantee transaction integrity via a double-entry ledger, enforce strict API quotas, and communicate with the outside world through a fully decoupled, async webhook pipeline.

🧠 Architecture & Engineering Decisions

Here's what actually matters — the hard systems problems this project solves.

1. Immutable Multi-Tenant Double-Entry Ledger

The problem: Storing balances as mutable integers (balance = balance + x) leads to race conditions, lost updates, and zero auditability in financial systems.

The solution: An append-only double-entry ledger. Every financial event generates immutable, balanced CREDIT and DEBIT entries bounded by a strict PostgreSQL transaction. Balances are derived, never overwritten. Strict tenant_id isolation guarantees zero cross-contamination of ledger state.

2. Zero-Bypass Rate Limiting

…

View on GitHub

DEV Community: Sidharth Yadav