DEV Community: Payneteasy

Cross-border payment reconciliation: matching multi-currency, multi-acquirer settlement files

Payneteasy — Fri, 05 Jun 2026 09:44:44 +0000

TL;DR

Reconciliation is the part of a payments stack nobody architects for on day one and everyone pays for on day 200.

The job: prove that every internal transaction matches the acquirer's settlement file, in the right currency, with the right fees, on the right value date — or surface the diff fast.

The mechanics: normalize files → land into an events table → project to a read model → diff against the internal read model → buckets for ops to resolve.

The boring details (file formats, fee parsing, FX rounding, value dates) are where 90% of the work lives.

If you've ever opened a CSV from an acquirer at the end of the month, sorted by amount, and tried to "just match it in Excel" — yes, this post is for you.

What "reconciled" actually means

A transaction is reconciled when, for the same logical payment, three views agree:

What you sent — your internal record of the charge/payout (your read model).
What the acquirer says happened — their settlement file or API report.
What the bank actually credited / debited — the bank statement.

Disagreements are normal. Persistent disagreements are how you lose money slowly and never know.

The shape of a settlement file

Across the major acquirers, settlement files look broadly similar — and broadly different in the places that matter:

Field	Variants you'll see
Transaction reference	acquirer's `transaction_id`, sometimes plus a `merchant_reference` round-tripped from you
Gross amount	minor units / decimal; transaction currency vs settlement currency
Fees	inline per-row, or aggregated at the file footer, or in a separate fees file
FX	inline rate vs separate FX file; sometimes only the converted amount
Value date	when the bank actually moves money — often T+1/T+2 from event date
Adjustments	refunds, chargebacks, fee corrections, reserves — usually mixed in
Encoding	UTF-8 if you're lucky; CP1252 / fixed-width / SWIFT MT940 if you're not
Granularity	one row per transaction or daily aggregates per merchant or both

There's no industry-clean schema for this. Plan to write one normalizer per acquirer.

Normalizing into events

The trick that pays off: don't try to reconcile files. Reconcile events, all in the same shape, in one table. Each row in the settlement file becomes an event:

CREATE TABLE settlement_events (
  event_id        uuid PRIMARY KEY,
  source          text NOT NULL,           -- acq_a / acq_b / bank_x
  source_file     text NOT NULL,           -- filename for traceability
  type            text NOT NULL,           -- charge / refund / chargeback / fee / fx_adjustment / reserve
  external_id     text NOT NULL,           -- acquirer's transaction id
  merchant_ref    text,                    -- your charge_id if they round-tripped it
  gross_minor     bigint NOT NULL,
  fee_minor       bigint NOT NULL DEFAULT 0,
  net_minor       bigint NOT NULL,
  currency        text NOT NULL,
  settlement_ccy  text NOT NULL,
  fx_rate         numeric(18,8),
  value_date      date NOT NULL,
  ts_event        timestamptz NOT NULL,
  raw             jsonb NOT NULL
);
CREATE UNIQUE INDEX ON settlement_events (source, external_id, type, value_date);

The raw column is a lifeline: keep the original row as JSON for every event. The first time the normalizer is wrong, you'll need to re-parse without re-downloading.

Two read models

Project to two flat tables — one from your internal events, one from the settlement events:

CREATE TABLE recon_internal (
  charge_id    uuid PRIMARY KEY,
  external_id  text,
  gross_minor  bigint NOT NULL,
  currency     text NOT NULL,
  fee_minor    bigint NOT NULL DEFAULT 0,
  net_minor    bigint NOT NULL,
  acquirer     text,
  status       text NOT NULL
);

CREATE TABLE recon_settled (
  charge_key     text PRIMARY KEY,         -- (acquirer, external_id) or fallback (acquirer, merchant_ref)
  acquirer       text NOT NULL,
  external_id    text NOT NULL,
  gross_minor    bigint NOT NULL,
  fee_minor      bigint NOT NULL,
  net_minor      bigint NOT NULL,
  currency       text NOT NULL,
  settlement_ccy text NOT NULL,
  fx_rate        numeric(18,8),
  value_date     date NOT NULL
);

Now reconciliation is a SQL query against two tables, not a script against a CSV.

The diff

WITH paired AS (
  SELECT i.charge_id,
         i.external_id,
         i.acquirer,
         i.gross_minor      AS internal_gross,
         s.gross_minor      AS settled_gross,
         i.fee_minor        AS internal_fee,
         s.fee_minor        AS settled_fee,
         i.currency         AS internal_ccy,
         s.currency         AS settled_ccy,
         s.value_date
  FROM   recon_internal i
  FULL OUTER JOIN recon_settled s
    ON s.acquirer = i.acquirer AND s.external_id = i.external_id
)
SELECT *,
       CASE
         WHEN charge_id IS NULL                         THEN 'unknown_in_settlement'
         WHEN external_id IS NULL                       THEN 'missing_settlement'
         WHEN internal_ccy <> settled_ccy               THEN 'currency_mismatch'
         WHEN internal_gross <> settled_gross           THEN 'gross_mismatch'
         WHEN internal_fee   <> settled_fee             THEN 'fee_mismatch'
         ELSE 'ok'
       END AS bucket
FROM paired
WHERE charge_id IS NULL OR external_id IS NULL
   OR internal_gross <> settled_gross
   OR internal_fee   <> settled_fee
   OR internal_ccy   <> settled_ccy;

Six buckets, six runbooks. Ops can drain each bucket independently. The numbers in each bucket are the only health metric reconciliation actually has.

Where the fiddly bits live

The query above is the easy part. The work is everywhere else.

Multi-currency

Two currencies in every settlement row: transaction currency (what the customer paid) and settlement currency (what the acquirer pays you in). Fees can be in either. FX may be inline or in a separate file with a value-date join. Two production rules:

Store both currencies and the rate. Never store only the converted amount.
Use a rounding rule chosen once and never argued with again — banker's rounding (half-to-even) is the safe default for finance.

Partial settlements and split files

A high-volume acquirer often splits a day's traffic across multiple files, sometimes across multiple days. Match on external_id, not file boundaries.

Fees that arrive separately

A few acquirers send a fees file independently of transactions. Project both into settlement_events with different types, then aggregate per external_id in the read model. Don't try to keep fees in a parallel structure.

Refunds and chargebacks

Each one is a new event, not an update to the original. They have their own external_id from the acquirer; they reference the original via a parent field that — surprise — is named differently per acquirer (original_transaction_id / refund_for / linked_id). Normalize on ingest.

Value-date drift

The transaction settles in your read model on T+0; the bank credits you on T+2. The reconciliation report can't blow up just because today's file doesn't include yesterday's late-evening charges. Compare by value date window, not "today".

A reusable matching heuristic

When external_id is missing or wrong (it happens), fall back through a small ladder:

def match(charge, candidates):
    # 1. exact match on external_id (best)
    by_ext = next((c for c in candidates if c.external_id == charge.external_id), None)
    if by_ext: return by_ext

    # 2. amount + currency + day + acquirer + last4
    for c in candidates:
        if (c.acquirer == charge.acquirer
            and c.gross_minor == charge.gross_minor
            and c.currency == charge.currency
            and abs((c.value_date - charge.event_date).days) <= 2
            and c.last4 == charge.last4):
            return c
    return None

The last4 is rarely on the settlement row but often on the transaction-detail report; if you can join the two, you've covered ~95% of "missing external_id" rows.

Metrics for a reconciliation function

If you're running this, three numbers belong on the dashboard:

Metric	Definition	Target
Match rate (T+1)	% of internal events with a matching settlement event by T+1 close	> 99% steady-state
Unmatched aging	oldest unmatched row's age, per bucket	days, not weeks
Net delta (per currency, per acquirer)	sum(internal_net) − sum(settled_net) over a day	within rounding

A reconciliation team's job isn't to do matching — it's to keep these three numbers honest while the engineers fix whichever normalizer is currently lying.

Closing

The settlement file is the truth your books are built on; treat it as a stream of events and the work scales. Treat it as a CSV to wrestle with and the work explodes.

If you're at the point where reconciliation is taking real engineering hours per month, the global payouts solution page lays out the model we use end-to-end (events → read models → diffs → ops queues).

Next in this series: what Verification of Payee and DORA change in your payment integration before the Sept/Jan 2026 deadlines.

*Author: payments engineer at PaynetEasy — we build payment orchestration and global payouts infrastructure → [payneteasy.com](https://payneteasy.com

Designing a payout ledger as a real-time read model (event sourcing in payments)

Payneteasy — Tue, 26 May 2026 20:48:44 +0000

TL;DR

A payouts table that gets UPDATEd on every webhook is the bug factory. Replace it with an append-only event log + a read model projected from it.

The read model is your "current balance / current state" view — rebuildable any time, idempotent, and cheap to query.

Reconciliation against bank/PSP statements becomes a diff between two read models, not a Friday-night batch.

Costs are real (storage, projection latency, schema discipline) but smaller than the cost of every payouts bug you'd otherwise pay.

If your payouts table looks like (id, status, amount, updated_at) and your code is full of WHERE status IN ('queued', 'in_flight', 'settled', 'failed'), this post is a refactor sketch worth keeping.

The CRUD-payouts trap

Most payout systems start the same way:

CREATE TABLE payouts (
  id           uuid PRIMARY KEY,
  user_id      uuid NOT NULL,
  amount_cents bigint NOT NULL,
  currency     text NOT NULL,
  status       text NOT NULL,         -- queued / in_flight / settled / failed / reversed
  acquirer     text,
  external_id  text,
  updated_at   timestamptz NOT NULL
);

The bugs follow within a quarter:

Webhooks arrive out of order — a settled overwrites a later reversed, and the user-facing balance is wrong.
Race between worker and webhook — both try to update status at once; last write wins, and last write is often the stale one.
No history — "why does this payout say failed?" has no answer beyond a Slack thread.
Reconciliation is a batch job — finance can't trust the table mid-day, so they wait for an overnight reconciliation that ends up patching the table from the bank statement.

Every one of these is structural. Updating a single row with the latest status throws away the truth.

Events as the source of truth

The truth is the sequence of things that happened. Model that, project everything else from it.

CREATE TABLE payout_events (
  event_id     uuid PRIMARY KEY,         -- idempotency
  payout_id    uuid NOT NULL,
  type         text NOT NULL,            -- created / submitted / settled / failed / reversed / fee_charged
  payload      jsonb NOT NULL,
  ts_event     timestamptz NOT NULL,     -- when it happened (PSP/bank time)
  ts_recv      timestamptz NOT NULL,     -- when we received it
  source       text NOT NULL             -- internal / psp_a / bank_statement / ops
);

CREATE INDEX ON payout_events (payout_id, ts_event);

A few conscious choices:

event_id is the idempotency key. Same event from the same source arrives twice → second insert is a primary-key conflict → no-op. This is how out-of-order delivery becomes safe.
Two timestamps. ts_event is the real clock (what the PSP says happened); ts_recv is your clock (when you saw it). Projections must order by ts_event and break ties deterministically. You'll thank yourself the first time a webhook arrives 6 hours late.
source tagged on every row. When the PSP and the bank statement disagree, you need to know which one said what.

Examples of events you'd actually emit:

{ "type": "payout.created",   "payout_id": "p1", "payload": { "amount_cents": 12000, "currency": "EUR", "user_id": "u1" } }
{ "type": "payout.submitted", "payout_id": "p1", "payload": { "acquirer": "acq_a", "external_id": "tx_998" } }
{ "type": "payout.settled",   "payout_id": "p1", "payload": { "external_id": "tx_998", "fee_cents": 35 } }
{ "type": "payout.reversed",  "payout_id": "p1", "payload": { "reason": "wrong_account" } }

The read model — what apps actually query

Apps don't want to fold over event logs. Project to a flat table that's fast to read and trivially rebuildable.

CREATE TABLE payout_state (
  payout_id    uuid PRIMARY KEY,
  user_id      uuid NOT NULL,
  amount_cents bigint NOT NULL,
  currency     text NOT NULL,
  status       text NOT NULL,        -- derived
  fee_cents    bigint NOT NULL DEFAULT 0,
  acquirer     text,
  external_id  text,
  last_event   uuid NOT NULL,        -- pointer back into payout_events
  last_ts      timestamptz NOT NULL
);

The projector is a small, deterministic function:

def project(state, event):
    t = event["type"]
    if t == "payout.created":
        return {
            "payout_id": event["payout_id"],
            "user_id":   event["payload"]["user_id"],
            "amount_cents": event["payload"]["amount_cents"],
            "currency": event["payload"]["currency"],
            "status": "queued",
            "fee_cents": 0,
            "last_event": event["event_id"],
            "last_ts": event["ts_event"],
        }
    if t == "payout.submitted":
        state.update(status="in_flight",
                     acquirer=event["payload"]["acquirer"],
                     external_id=event["payload"]["external_id"])
    elif t == "payout.settled":
        state.update(status="settled",
                     fee_cents=state["fee_cents"] + event["payload"].get("fee_cents", 0))
    elif t == "payout.failed":
        state.update(status="failed")
    elif t == "payout.reversed":
        state.update(status="reversed")
    state["last_event"] = event["event_id"]
    state["last_ts"]    = event["ts_event"]
    return state

Three properties of a good projector:

Pure. No side effects beyond writing the state row. No external calls, no clocks.
Idempotent. Replaying the same event must produce the same result. The event_id short-circuit lives at the caller layer.
Order-tolerant. A late submitted arriving after settled shouldn't downgrade the status. Encode the rule explicitly:

RANK = {"queued": 0, "in_flight": 1, "failed": 2, "settled": 3, "reversed": 4}

def upgrade(state, candidate_status):
    if RANK[candidate_status] >= RANK[state["status"]]:
        state["status"] = candidate_status

That's the entire "out-of-order webhooks" problem solved in five lines.

Rebuilding the read model

The thing event sourcing buys you that the CRUD table can't: you can always rebuild.

def rebuild(payout_id):
    state = {}
    for ev in events_for(payout_id, order_by="ts_event"):
        state = project(state, ev)
    upsert("payout_state", state)

In production you run this on:

Schema migrations. Add a new field to payout_state? Backfill by replay, not by UPDATE ... SET.
Bug fixes in the projector. A wrong status mapping fixes itself on replay.
Reconciliation discrepancies. When a balance is off, dump the events, replay, and the bug is either in the events (missing/extra) or in the projector (logic). One of those two — never both.

Reconciliation as a diff

This is where the model earns the rest of its weight. Take the bank statement, ingest it as events (source = bank_statement), project to a separate read model, and diff:

SELECT s.payout_id, s.amount_cents AS internal, b.amount_cents AS bank
FROM   payout_state s
FULL OUTER JOIN payout_state_bank b USING (payout_id)
WHERE  s.amount_cents IS DISTINCT FROM b.amount_cents
   OR  s.status       IS DISTINCT FROM b.status;

The output is a finite list of disputes-with-the-truth. Each row is either:

A missing event on one side (most common) — backfill from the other source.
A real money discrepancy (rare, important) — escalate.

Note the asymmetry: you don't correct the read model directly. You add the missing event, replay the projector, and the diff resolves itself. That's the discipline that keeps the books honest.

What it costs you

Honest tradeoffs:

Cost	Mitigation
More storage (events live forever)	Cold storage / archive tier after N months; events are tiny
Projection latency (a few ms → seconds for high volume)	Synchronous projector for the common path; async for bulk events
Schema discipline (event types are forever)	Treat event types like API contracts — versioned, deprecated, never silently changed
Operational learning curve	One page in the runbook on "how to replay" pays for itself the first incident

These are real but not large. The alternative is debugging a payouts table at midnight every settlement window.

What about your accounting ledger?

Two ledgers, two purposes:

The payout read model is for operational state — what's in flight, what settled, what failed. Cached, fast to query, regenerable.
The accounting ledger (debits/credits, double-entry) is for financial state — what your books look like. Driven from the same events but with stricter rules and an immutable history that finance can audit.

Don't confuse the two. Both should be projections from the same event log.

Closing

Event sourcing isn't free, but for payouts specifically the alignment is unusually clean: payments are events in the real world, and webhooks are the world telling you about them. The read model is the part your apps query; the events are the truth you can always fall back to.

If you want to see how this fits the wider orchestration picture — global payouts, multi-acquirer settlement, FX, reconciliation — the global payouts solution page has the architecture diagram.

Next in this series: cross-border payment reconciliation — matching multi-currency, multi-acquirer settlement files against the read model above.

Author: payments engineer at PaynetEasy — we build payment orchestration and global payouts infrastructure → payneteasy.com

Smart transaction routing: turning auth-rate data into routing rules (without a black box)

Payneteasy — Wed, 20 May 2026 09:11:49 +0000

TL;DR

A "smart router" is not a model — it's a rules engine fed by fresh, segmented approval-rate data.

Inputs you actually need: BIN/issuer, country, MCC, amount, currency, retry-count, acquirer health, rolling approval window.

Cascading retries: retry only soft declines, never hard / fraud / lost-stolen. Fresh idempotency key per attempt.

Approval-rate-aware routing without a circuit breaker is a way to hammer a dead acquirer. Don't skip the breaker.

The phrase "smart transaction routing" gets used to sell a lot of black boxes. This post is the opposite: how to build one whose decisions a human can read, and how to keep it honest with data.

Why a single acquirer stops being enough

Three things, sooner or later:

A regional approval dip. Card schemes adjust, an issuer changes its fraud rules, you wake up to a 5pp drop in DE/NL and zero levers to pull.
An outage. Acquirers go down. A merger or migration takes a network offline for a window. With one acquirer, your checkout is down with it.
Pricing leverage. Once you can move a percentage of traffic with a config change, every renegotiation is real.

The fix is structural: route every transaction through a layer that knows about more than one acquirer and decides where it goes.

Anatomy of a rules engine

Inputs that matter (in roughly this order of usefulness):

Input	Why it changes the route
Issuer BIN / country	Local acquirers approve local cards better — almost universally
Currency / amount	Cross-border fees jump at currency mismatches and ticket size
MCC	Some acquirers are strong in specific verticals (travel, digital goods, subscription)
Brand (Visa / Mastercard / Amex / local)	Amex / domestic networks often need a specialist acquirer
Retry count	Sticky-on-retry vs failover-on-retry are different strategies
Acquirer health / approval-rate window	Excludes acquirers in an active dip
Time-of-day / day-of-week	Some issuers have approval cycles — rarely worth bothering with day one

Outputs are simple: the chosen acquirer (plus credentials) and a fallback chain.

# rules.yaml
- name: eu-cards-primary
  match: { issuer_country: [DE, FR, NL, ES], currency: EUR, amount_lt: 50000 }
  route:
    - { acquirer: acq_a, weight: 70 }
    - { acquirer: acq_b, weight: 30 }
  fallback: [acq_c]

- name: high-ticket-amex
  match: { brand: amex, amount_gte: 50000 }
  route: [{ acquirer: acq_amex_specialist, weight: 100 }]
  fallback: [acq_a]

- name: latam-default
  match: { issuer_country: [BR, MX, CO, AR] }
  route: [{ acquirer: acq_local_latam, weight: 100 }]
  fallback: [acq_a]

def evaluate(rules, ctx):
    for r in rules:
        if matches(r["match"], ctx):
            return pick_weighted(r["route"]), r.get("fallback", [])
    return default_route(ctx), default_fallbacks()

The rules file is the contract between engineering and ops. If your PM can't read it without you, you've built a black box. The most common mistake here is letting the matcher grow regex/DSL features until only its author understands it — resist.

Routing strategies, with honest tradeoffs

Strategy	When to use	Optimizes	Risk
Least-cost	Stable approval rates across acquirers	Fees per approved tx	A penny saved on fees can cost a dollar in declines if approval is uneven
Approval-rate-aware	Volatile approval, multi-region	Overall approval %	Requires fresh data; flap risk if the window is too small
Weighted A/B	Onboarding a new acquirer	Risk-controlled ramp	Don't keep the A/B forever — pick a winner
Sticky-on-retry	Card-on-file retries	Consistency / fewer step-ups	Sticky to a failing acquirer = obvious bug
Failover-on-retry	First attempt failed, try elsewhere	Recovers approvals	Wrong on hard declines — see below

The boring answer is that mature stacks combine all of these — the rules file becomes the explicit place where you say when each one fires.

Turning auth-rate data into a routing rule (without a model)

You don't need ML for this on day one. A rolling window per (acquirer, bin_country, brand) is enough to be useful:

def approval_rate(acq, bin_country, brand, window=timedelta(minutes=15)):
    rows = stats.fetch(acq, bin_country, brand, since=now()-window)
    if rows.attempts < MIN_SAMPLE:
        return None  # not enough signal — fall back to the default rule
    return rows.approved / rows.attempts

Two production-grade details:

Minimum sample. With low volume on a corridor, a single decline drops the rate to 0% and you'd eject a fine acquirer. Require N ≥ 50 attempts before letting the data steer.
Damping. Don't switch winners on every refresh. EMA, hysteresis bands, or a 5-minute lockout after a flip — pick one.

Then the routing rule becomes:

def best_acquirer(candidates, ctx):
    ranked = []
    for acq in candidates:
        rate = approval_rate(acq, ctx.bin_country, ctx.brand)
        if rate is None: rate = baseline(acq, ctx)
        if rate < KILL_SWITCH:           # e.g. 0.10
            continue
        ranked.append((rate, acq))
    ranked.sort(reverse=True)
    return [acq for _, acq in ranked]

That's the entire "smart" of smart routing on day one. Add cost-weighting after the approval-rate signal is stable.

Cascading retries done right

Retry soft declines, never hard ones:

Decline class	Retry?	Why
`do_not_honor` (often issuer transient)	Yes — different acquirer	Issuers re-evaluate via a different fingerprint
`insufficient_funds`	Maybe, after delay	Topping-up is a real event but most retries are wishful
`issuer_unavailable` / `network_timeout`	Yes	Definitionally transient
`lost_card` / `stolen_card` / `pickup_card`	Never	Card-network rule; you'll get fined
`do_not_honor` flagged as fraud	Never	Fraud scores stack
`expired_card`	No	Need new credentials

def charge_with_cascade(ctx, primary, fallbacks):
    for acq in [primary, *fallbacks]:
        if circuit_open(acq):
            continue
        res = adapters[acq].charge(ctx, idem_key=attempt_key(ctx))
        if res.approved: return res
        if res.taxonomy in ("hard", "fraud"):
            return res   # do NOT cascade
    return Declined("all routes exhausted")

The two non-obvious details:

Fresh idempotency key per attempt. Different acquirers don't share state; reusing the same key is undefined.
Hard-decline short-circuit. This is also the thing that protects you from disputes if a card was reported stolen between attempts.

Health checks, circuit breakers, observability

The router will lie to you if it's not measured. The minimum kit:

Synthetic pings. Heartbeats per acquirer, decoupled from real traffic.
Error-rate circuit breaker. Trip on (5xx + timeouts) / total > X% over Y minutes. Auto-eject, auto-return after a cool-down.
Approval-rate alerting. Page on a 3σ drop per (acquirer × bin_country).
Per-rule shadow. Log what the previous rule version would have decided. You'll need this every time you change a rule.

@dataclass
class CircuitBreaker:
    fail_rate_threshold: float = 0.25
    window: timedelta = timedelta(minutes=5)
    cooldown: timedelta = timedelta(minutes=10)

    def open_for(self, acq) -> bool: ...
    def record(self, acq, ok: bool) -> None: ...

Metrics that matter

Metric	Formula	Target
Overall approval %	approved / attempted (unique tx)	maximize
Cost per approved tx	total fees / approved	minimize
Fallback rate	tx using fallback / total	low + stable
Retry success uplift	extra approvals from cascade / attempted	track & celebrate
p95 routing latency	—	< 50 ms
Per-rule decision drift	rules diff vs shadow	review weekly

Without these you have a router; with them you have a routing system.

Build vs buy

This is the article that gets answered most often with "buy" — and that's usually right when routing isn't your moat. Where an orchestration platform actually saves months is in the adapters, normalized error taxonomy, and reconciliation, not the rules engine itself. The rules engine is the easy part; staying current with 12 acquirer APIs is what burns a team out.

A pragmatic split: keep the rules and the data in-house, integrate the adapters via a platform. That gives you the strategic edge and removes the busy-work.

If you want the orchestration-layer view of the whole picture, the payment orchestration overview walks through the architecture and the build-vs-buy lines we use with customers.

The next post in this series gets into cross-border reconciliation — settlement files, currency, fee parsing — which is where most homemade routing layers quietly fall over.

*Author: payments engineer at PaynetEasy — we build payment orchestration and global payouts infrastructure → payneteasy.com

What's your idempotency-key TTL — and what broke when you got it wrong?

Payneteasy — Mon, 18 May 2026 22:26:17 +0000

Quick one for the payments / API engineers in here.

Idempotency keys are one of those features where "we have them" hides a lot of variance. The piece nobody agrees on is the TTL.

I've seen production systems run with:

1 hour — fine for browser checkout, breaks the moment a mobile app retries from an offline queue 2 hours later.
24 hours — the default-ish answer, fits an end-of-day reconciliation cycle.
48–72 hours — server-to-server with human-in-the-loop retries.
7 days — usually because someone got burned once and didn't want to pick a new fight with finance.

The honest heuristic I keep coming back to: TTL ≥ your longest legitimate retry path, and < your reconciliation window. Anything tighter and a real retry lands after the key has expired (double charge); anything looser and you can't close the books cleanly.

So — two questions, looking for war stories more than theory:

What TTL do you run, and what's the use case it's tuned for?
Have you ever had a duplicate transaction (or a near-miss) because the TTL was wrong? What did the post-mortem land on?

(If you're using idempotency keys without a body fingerprint, that's a different conversation, and worth its own thread — same key + different body is a quiet way to get the wrong thing accepted.)

I'm collecting answers for a follow-up post on idempotency patterns in payment APIs — happy to credit anyone who shares a story. Comments below 👇

Author: payments engineer at PaynetEasy — we build payment orchestration and global payouts infrastructure.

Idempotency keys in payment APIs: the patterns that actually prevent double charges

Payneteasy — Sun, 17 May 2026 09:58:53 +0000

TTLs, key derivation, retry storms, and the six ways teams accidentally let the same charge through twice.

TL;DR

The idempotency key is the only thing that keeps "the network blinked, let's retry" from turning into two real charges.

The key has to be client-generated, stable across retries, scoped to the operation, persisted server-side, and TTL'd — miss any one and you have a bug waiting for a busy Friday.

Replay semantics matter: a retry with the same key must return the same response, not "OK, accepted" again.

The six classic ways teams get this wrong are all preventable. They just need to be on a checklist.

If your payment API has a /charge endpoint and no idempotency key in the request header, this post is for you. If it has one but you're not sure what the TTL should be — also for you.

The shape of the problem

Every retry in a payment system is a coin flip on a duplicate. The classic failure goes like this:

client ──POST /charge───▶ server  (200 OK, settled)
       ◀──── socket reset ────
client ──POST /charge (retry)──▶ server  (200 OK, settled AGAIN)

The server has no way of knowing the second request is the same one. The card was just charged twice. The customer files a chargeback. You eat the fee and (depending on the network) a hit to your dispute ratio.

An idempotency key fixes this with one rule: for a given key, the server returns the same response for a window of time, regardless of how many times the request arrives.

client ──POST /charge   key=k1──▶ server  (200 OK, settled, store(k1, resp))
       ◀──── socket reset ────
client ──POST /charge   key=k1──▶ server  (returns stored resp; no new charge)

Five properties of a key that works

Client-generated. The point of the key is to survive a retry; if the server generates it, every retry gets a new one. Use a UUIDv4 or <order_id>:<attempt> — anything the client controls.
Stable across retries. The same logical operation must reuse the same key for the entire retry window. If you regenerate on each attempt, you've defeated the mechanism.
Scoped to the operation. "Charge $42" and "Refund $42" share nothing; reusing a key across them is undefined behavior. Most APIs scope by (merchant, endpoint, key).
Persisted server-side. Not in memory, not in the load balancer — in a durable store the next worker can read. Otherwise a process restart between attempt #1 and attempt #2 nukes the protection.
TTL'd. Keys live for a bounded window (commonly 24h). After that they expire so you can reuse them for new operations and your store doesn't grow forever.

A reference implementation

import hashlib, json, time
from typing import Optional

TTL_SECONDS = 24 * 3600   # 24h — fits a typical end-of-day reconciliation window

def fingerprint(body: dict) -> str:
    canonical = json.dumps(body, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode()).hexdigest()

def handle_charge(req, store, acquirer):
    key   = req.headers.get("Idempotency-Key")
    if not key:
        return 400, {"error": "Idempotency-Key required"}

    fp = fingerprint(req.body)
    cached = store.get(key)

    if cached:
        # Same key, same body → replay the previous response
        if cached["fingerprint"] != fp:
            return 422, {"error": "Idempotency-Key reuse with different body"}
        return cached["status"], cached["response"]

    # First sighting — reserve the slot so concurrent retries serialize
    if not store.set_if_absent(key, {"state": "in_flight", "fingerprint": fp, "ts": time.time()}, ttl=TTL_SECONDS):
        # Another worker is already processing this key — wait or return 409
        return 409, {"error": "request in progress"}

    res = acquirer.charge(req.body)
    store.set(key, {"state": "done", "fingerprint": fp, "status": 200, "response": res, "ts": time.time()}, ttl=TTL_SECONDS)
    return 200, res

Three details worth flagging:

The fingerprint check (cached["fingerprint"] != fp) is what turns "I sent the same key twice with different amounts" from an exploit into a 422.
The set_if_absent reservation is what stops two concurrent retries from both calling the acquirer at once. Without it, you've moved the race condition one layer down.
The 24h TTL is a convention, not a law. Pick it to match your operational window — see the next section.

TTL: how long is long enough?

This is where teams disagree. The answer depends on what your retry surface looks like.

Caller	Realistic retry horizon	TTL guidance
Browser checkout	seconds → a few minutes	1h is plenty
Mobile app (offline-tolerant)	up to 24h (offline queue)	24h
Server-to-server with manual retry	up to a few days	48–72h
Batch payouts (overnight files)	aligned to settlement cycle	match the cycle (often 24h)

Two heuristics that survive most arguments:

Longer than your longest real retry path, shorter than your reconciliation cycle. If a retry can land after you've closed the books on that day, your books are wrong.
If in doubt, 24h. Then revisit when you have data on actual key reuse age.

The six classic ways teams get this wrong

Regenerating the key on each retry. Almost always a bug in a retry library's "what changes per attempt" config. Fix: generate the key once, persist it with the order, reuse it.
Storing keys in memory only. Works until your process restarts. Fix: durable store (Postgres unique-index, Redis with persistence, DynamoDB).
Not scoping the key. A refund retry uses the same key as the original charge → 422 or, worse, undefined. Fix: scope by (merchant, endpoint, key).
Not fingerprinting the body. Same key + different body = silent acceptance of whichever request arrived first, with no way to tell. Fix: hash the canonical body and compare.
No reservation between concurrent retries. Two retries arrive on two workers, both miss the cache, both call the acquirer. Fix: INSERT ... ON CONFLICT DO NOTHING or a Redis SET NX.
TTL shorter than the retry window. Mobile app retries 2 hours later; the key has expired; the charge goes through twice. Fix: TTL ≥ longest legitimate retry path.

Idempotency in cascading retries

In a multi-acquirer setup, "retry" can mean two different things:

Same acquirer, transient network error → same idempotency key. You want the acquirer to deduplicate.
Different acquirer, after a hard failure → new idempotency key per attempt. The acquirers don't share state; reusing the same key would either be ignored or — worse — happen to clash with someone else's traffic.

A clean way to encode this:

def attempt_key(charge_id: str, attempt: int) -> str:
    return f"{charge_id}:a{attempt}"   # client-stable, per-attempt

charge_id is the logical operation (lives in your DB, survives retries), attempt increments only when you fall over to a different acquirer.

A small checklist for your code review

[ ] Header name documented (Idempotency-Key) and required on mutating endpoints.
[ ] Server returns 400 if missing on charge/refund/payout endpoints.
[ ] Body fingerprint stored alongside the key; mismatch = 422.
[ ] Reservation pattern prevents concurrent execution.
[ ] TTL ≥ your longest legitimate retry path and < your reconciliation window.
[ ] Replay returns the original status code, not a generic 200.
[ ] Tested with a fault injector that drops responses between server commit and client receipt.

If your checkout endpoint passes all seven, you've taken the most common cause of double charges off the table.

The next post in this series treats the payout ledger as a real-time read model — same correctness mindset, different shape: append-only events, idempotent projections, end-of-day reconciliation that doesn't need a Friday-night batch. If you want the orchestration-layer view, see the payment orchestration overview.

*Author: payments engineer at PaynetEasy — we build payment orchestration and global payouts infrastructure → payneteasy.com

Payment orchestration for engineers: what it is, when you actually need it, and build-vs-buy

Payneteasy — Thu, 14 May 2026 10:39:29 +0000

Not a buzzword. Orchestration is a routing layer + a normalized payment model + a reconciliation spine. Here's the engineering view — and an honest build-vs-buy.

Payment orchestration is three things, not one: a provider-agnostic API, a routing layer (retries, cascades, least-cost / best-auth-rate decisions), and a reconciliation spine that ties every attempt back to money that actually moved.
You don't need it on day one. You need it the day your codebase grows its third if provider == "X" branch — or the day Finance asks "why don't these two reports match?"
"We'll just add acquirer #2 ourselves" is the classic trap: the easy 20% (a second adapter) is visible; the hard 80% (idempotency across providers, webhook fan-in, one settlement model, dispute plumbing) shows up six months later.
Build-vs-buy is a real decision with real numbers. This post gives you the component checklist, a provider-agnostic interface sketch, a decision matrix, and a migration path that doesn't bet the checkout on a big-bang cutover.

The symptom: your payment code is branching on the provider

Here's the smell. Somewhere in your codebase there's a function that started innocent:

def charge(order, provider):
    if provider == "acquirer_a":
        return acquirer_a.create_payment(order.amount, order.currency, order.card)
    elif provider == "acquirer_b":
        # different field names, cents vs decimal, different 3DS flow...
        return acquirer_b.charge({"sum": order.amount * 100, "cur": order.currency, ...})
    elif provider == "wallet_x":
        # redirect-based, no card at all, async callback
        ...

Every new payment method or geography adds a branch. The branches leak into refunds, into webhook handlers, into reconciliation, into your fraud checks. Soon "add a payment provider" is a two-sprint project and nobody wants to touch the file.

That branching is the thing orchestration removes. Not by magic — by forcing a normalized payment model at the boundary and pushing every provider's quirks into an adapter behind it.

If you have exactly one PSP and no plans to add another, you don't have this problem yet. Don't build orchestration for a problem you don't have. The rest of this post is about recognizing when you do have it.

What "orchestration" actually is — the five components

Strip the marketing and a payment orchestration layer is five concrete things. If a "platform" gives you only the first one, you've bought a thin proxy, not orchestration.

Component	What it owns	What breaks without it
1. Normalized payment model	One `Charge` / `Refund` / `Payout` shape; one currency representation; one status vocabulary; one error taxonomy	Provider quirks leak into business logic; every consumer re-learns each provider
2. Routing layer	Which provider gets this transaction; retries; cascades on soft declines; least-cost / best-auth-rate decisioning; circuit breakers	One acquirer outage = checkout down; no recovery of "issuer was flaky for 400ms" declines
3. Retry & cascade engine	Idempotency keys across providers; bounded re-attempts; decline-code awareness; resume-after-3DS	Double charges; infinite retry storms; cascading a `stolen_card` (don't)
4. Webhook fan-in	Normalizing N providers' async callbacks into one event stream; de-dup; ordering; replay	You write N webhook handlers, each with its own retry semantics; missed/duplicate events
5. Reconciliation spine	Joining attempts → authorizations → captures → settlement files → ledger; flagging mismatches	Finance reports that don't tie out; disputes you can't evidence; silent revenue leakage

Components 1–3 are what most people picture. Components 4 and 5 are where the real engineering is, and they're the ones a DIY "we just added an adapter" effort almost always skips. A second adapter is a week. A reconciliation spine that survives a contested chargeback eight months later is not.

Code: the provider-agnostic boundary

The heart of component 1 is an interface every provider adapter implements. Keep it small and money-shaped:

// The normalized model — providers adapt TO this, not the other way around.
type Money = { amountMinor: bigint; currency: string }; // always minor units, no floats

interface PaymentProvider {
  readonly id: string;
  authorize(req: AuthorizeRequest): Promise<AuthorizeResult>;
  capture(authId: string, amount: Money, idemKey: string): Promise<CaptureResult>;
  refund(captureId: string, amount: Money, idemKey: string): Promise<RefundResult>;
  // Async truth arrives via webhooks, normalized to the same event type:
  parseWebhook(raw: HttpRequest): NormalizedEvent | null;
}

type AuthorizeResult =
  | { status: "approved"; authId: string; providerRef: string }
  | { status: "declined"; code: DeclineCode; retryable: boolean }   // taxonomy is OURS
  | { status: "action_required"; kind: "3ds_challenge" | "redirect"; url: string }
  | { status: "error"; transient: boolean };                        // network/5xx ≠ decline

Two design rules that save you later:

Money is always minor units in an integer/bigint. No amount * 100 scattered across adapters. The conversion happens once, in the adapter, on the way in and out.
The decline-code taxonomy belongs to you, not to a provider. Each adapter maps that provider's 91 / do_not_honor / try_again_later onto your enum and sets retryable. The routing layer never sees a raw provider code.

And the routing layer that sits on top is, at its simplest, a scored candidate list — not a black box:

def pick_candidates(txn) -> list[Provider]:
    scored = []
    for p in eligible_providers(txn):           # currency, method, scheme, geography
        if circuit_open(p):                     # provider in cooldown? skip
            continue
        score = (
            0.55 * rolling_auth_rate(p, txn.bin_country, txn.amount_band)  # what works
          + 0.30 * (1 - normalized_cost(p, txn))                           # what's cheap
          + 0.15 * health_score(p)                                         # latency/errors
        )
        scored.append((score, p))
    return [p for _, p in sorted(scored, reverse=True)][:MAX_HOPS]   # bounded!

# Then: try candidate 0; on a *retryable* decline, cascade to candidate 1; stop at
# MAX_HOPS or a wall-clock deadline; never cascade a hard decline (insufficient_funds,
# stolen_card, ...). One idempotency key per *cascade*, reused per hop — see the
# idempotency-keys post in this series.

That's the whole "AI routing" mystique demystified for the simple case: features → score → ordered list → bounded cascade, with every decision logged so you can answer "why did this $4,000 transaction go to acquirer C?" A model can replace the weighted sum later; the explainability requirement doesn't go away when it does.

When a single PSP is fine — and when you've outgrown it

Adding orchestration has a cost (a new layer in your most critical path). Use it when the pain is real, not aspirational.

Situation	Single PSP is fine	You've outgrown it
Geographies	One market, local cards	Multiple regions, local methods (iDEAL, PIX, UPI, SEPA Instant…)
Volume	Low enough that a 0.5–2 pp auth-rate gap is noise	High enough that 1 pp of auth rate is a meaningful revenue line
Resilience	An hour of PSP downtime is survivable	PSP downtime = direct revenue loss / SLA breach
Cost	Interchange-plus is whatever it is	Routing by cost/scheme would save real money at your volume
Method sprawl	Cards (+ maybe one wallet)	A growing matrix of methods × providers, each with its own webhook
Org	One team owns payments end-to-end	Finance, fraud, and product all consume payment data and it must agree

A useful litmus test: count the if provider == branches and the distinct webhook handlers. One of each — you're fine. Three or more — orchestration will pay for itself, either as something you build deliberately or something you buy.

Build-vs-buy, honestly

This is a genuine fork, and the honest answer is "it depends — here's on what."

Dimension	Build it yourself	Buy / adopt a platform
Upfront engineering	~6–18 eng-months for a real one (model + routing + retries + webhook fan-in + reconciliation), not the 1-month adapter	Integration weeks, not months
Ongoing maintenance	Permanent: new provider quirks, scheme mandates (3DS, SCA, network tokens, VoP…), reconciliation edge cases	Mostly absorbed by the vendor; you track their changes
Compliance / PCI scope	You may pull more card data into scope unless you're careful with tokenization	Often reduces your scope (vaulting, redirect/iframe) — verify per vendor
Acquirer contracts	You negotiate and hold every contract	Either you bring your own acquirers (BYO-acquiring) or use theirs
Control & differentiation	Total control; routing logic can be a competitive edge	Less control over the deep internals; you depend on roadmap
The hidden 80%	Idempotency across providers, webhook ordering/de-dup, dispute evidence, settlement-file parsing, ledger truth	Should be solved already — make this an evaluation question, not an assumption

Rule of thumb: build it if payment routing is core to your product's economics or differentiation (you're a marketplace, a PSP, a platform whose margin lives in routing) and you can fund the ongoing team. Otherwise the maintenance tail — not the initial build — is what makes "buy" win. Either way, the component checklist above is your spec: if you buy, score vendors against all five; if you build, don't ship 1–3 and call it done.

Anti-patterns

Orchestration as a thin proxy. A normalized API in front of two providers with no reconciliation spine = you built the easy 20% and named it after the hard part. The first contested chargeback exposes it.
Routing without circuit breakers. "Best auth rate" routing that keeps hammering a provider mid-incident turns one provider's outage into your outage.
Per-attempt idempotency keys. Regenerating the key on each cascade hop defeats the point — a retried HTTP call to the same provider can now create a second authorization. One key per cascade.
Floating-point money. amount * 100 in three adapters with three rounding behaviors. Minor units, integers, one conversion site.
Measuring auth rate per attempt. Cascades make per-attempt auth rate look worse and routing changes look better than reality. Attribute per cascade.
Big-bang cutover. Moving 100% of checkout to a new orchestration layer over a weekend. Don't.

Migration path: strangler-fig, not big bang

If you're adding orchestration to a live system:

Wrap, don't replace. Put the normalized PaymentProvider interface in front of your existing PSP first. Same provider, new boundary. Ship that. Nothing user-visible changes.
Route a sliver. Send 1% of eligible traffic through the new layer (still to the same provider). Watch auth rate, latency, error rate, and — critically — that reconciliation still ties out.
Add provider #2 behind the interface. Now it's an adapter, not a branch in business logic. Route a small % to it; compare auth rates per BIN-country/amount-band.
Turn on cascades on retryable declines, bounded (MAX_HOPS, deadline). Measure recovered transactions and double-auth incidents (should be ~0, caught by the reconciliation sweep).
Move webhooks to the fan-in. One normalized event stream; retire the per-provider handlers.
Make the ledger the source of truth. Reconciliation runs against the spine, not against one provider's dashboard. Now adding provider #3 is a checklist, not a project.

Each step is independently shippable and independently reversible. That's the point.

Copy-this checklist

[ ] Normalized money type — minor units, integer/bigint, converted once per adapter.
[ ] Decline-code taxonomy is yours; each adapter maps the provider's codes onto it and sets retryable.
[ ] error (network/5xx/timeout) is not the same as declined in your model.
[ ] Routing produces a bounded candidate list (MAX_HOPS, wall-clock deadline).
[ ] Every routing decision is logged with its reason (explainable, even if a model picks).
[ ] Circuit breakers pull a provider out of rotation on error-rate spike; bleed traffic back gradually.
[ ] One idempotency key per cascade, reused per hop — never regenerated.
[ ] Webhook fan-in: de-dup, ordering, replay — one normalized event stream, not N handlers.
[ ] Reconciliation spine joins attempt → auth → capture → settlement file → ledger; flags mismatches.
[ ] Auth rate measured per cascade, not per attempt; you also watch cost per approved txn.
[ ] If buying: vendor scored against all five components, not just the normalized API.
[ ] If building: rollout is strangler-fig (wrap → 1% → provider #2 → cascades → webhooks → ledger), each step reversible.

*Written by the engineering team at PaynetEasy — payment orchestration & cross-border payouts infrastructure. We write about routing, reconciliation and money-movement correctness at payneteasy.com.