Idempotency and Retry in a Payment Core: Operations That Can't Be Duplicated and Can't Be Forgotten

Build operations that can't be duplicated and can't be forgotten — that's the contract for a payment core.

On the core I built for ROSSTRAFFY (20,000+ successful transactions/day, 99.99% uptime), the most dangerous failure class wasn't crashes. It was retries.

The network doesn't guarantee delivery. A request can be processed, but the response lost — the client retries, and the server can't tell "didn't arrive" from "processed, but response lost". If a retry means "run the request again", you get double charges. If it means "resume the same operation", you get exactly one outcome per key.

The contract: every payment happens exactly once — not 0, not 2 times. Zero means a lost payment and a missed discount window; two means double charges, refunds, disputes.

The approach: the payment core as a state machine

A payment operation is not a "request" that either ran or didn't. It's a path with state: created, processing, completed, rejected, in reconciliation. Every transition must be idempotent — applying the same transition twice must not change the result.

Idempotency lives at three levels, and skipping any one lets duplicates through:

1. API level — one idempotency key per business operation, generated once, surviving all retries.
2. Handler level — the operation is a state machine; every transition checks the current state before acting.
3. Storage level — unique constraints as the last line of defense.

What we built

• Operation = state machine: every transition idempotent.
• One idempotency key per business operation, generated once, surviving all retries.
• Idempotency at three levels: API, handler, storage (unique constraints as the last line of defense).
• Payment = two operations: hold (block funds) → capture (charge after the external system confirms), each with its own key and retry rules.
• Sync leg: bounded retries, exponential backoff, timeouts, jitter, dead-letter queue.
• Async leg (ГИС ГМП): queue + workers, load balancing across 5+ acquirers (Strategy pattern), failover on degradation → 99.8% acceptance reliability.
• Recurring payments stay on the same acquirer (confirmed token); switching only for new payments and emergencies.
• Observability: counters + latency per stage in Grafana, alerts on anomalies. "More retries, fewer successful operations" = retry is masking the problem, not fixing it.
• Reconciliations as the second safety net (ClickHouse: daily financial reports in 3–5 minutes instead of hours).

Validation and results

We validated with A/B tests behind feature flags on real traffic. Effects were fractions of a percent of conversion — only real payment statistics could see them; synthetic tests couldn't.

Metric	Result
Erroneous and fraudulent operations	−25%
Payment conversion	+~10%
Median transaction processing time	−15%

Takeaways

Idempotency isn't a nice-to-have in payments. It's the contract that makes retries safe at scale — and what turns "the user paid twice" into "the user paid once, and we can prove it".

The full article with diagrams and code (EN): https://frolov.guru/en/writing/idempotency-retry/

---

About the author: Alexander Frolov — Senior/Staff Backend Engineer (PHP, highload) and Team Lead, 18+ years: payment cores, multi-tenant platforms at federal scale. Articles on highload PHP and architecture: frolov.guru. If you're fighting timeouts, duplicates or lost payments — that's an idempotency problem, and it's fixable. DM or [e-mail].