Idempotency, vendor failure, monitoring that catches the invisible outages, and the tradeoffs nobody warns you about, lessons from scaling payments infrastructure.
Most software fails quietly. A page renders slowly, a recommendation is a little off, a report is stale by an hour. Users shrug and move on.
Payments doesn't work like that. When payments break, someone's money is in a place neither of you can account for, and the clock starts ticking on their patience. There's no graceful degradation. Either the money moved, or it didn't, and someone needs to know which.
I've spent a good chunk of my career building and scaling payments infrastructure, and it has quietly rewired how I think about engineering in general. Here's what stuck.
📋 The short version
| # | Lesson | One-line summary |
|---|---|---|
| 1 | Idempotency | You will receive the same request twice. Design for it. |
| 2 | Vendor failure | Gateways are vendors. Ask "when," not "if." |
| 3 | Monitoring | Never learn about an outage from a customer. |
| 4 | The unglamorous stuff | Ledgers, reconciliation, state machines, refunds. |
| 5 | Tradeoffs | Every lesson above fights at least one other. |
1. 🔁 Idempotency isn't a feature. It's a foundation.
The first hard lesson: you will receive the same request twice. Not "might." Will.
A client times out waiting for your response and retries. A user double-taps a button on a bad connection. A queue consumer crashes after processing but before acknowledging. A gateway sends the same webhook four times because it never got a 200 back. None of these are exotic failure modes, they're Tuesday.
If your system treats every incoming call as a new instruction, every one of those scenarios becomes a double charge. And a double charge isn't a bug you fix quietly in the next release. It's a support ticket, a refund, a reconciliation entry, and a customer who now checks their statement every time they use you.
The fix is conceptually simple and operationally demanding: every operation that moves money must be uniquely identifiable and safely repeatable. The caller supplies an idempotency key. You store it before you do anything else. If you see it again, you return the original result, not a new attempt, not an error, the same answer you gave the first time.
The naive version everyone writes first:
# ❌ Race condition hiding in plain sight
existing = db.find_by_idempotency_key(key)
if existing:
return existing.result
result = charge_customer(amount) # two concurrent requests
db.save(key, result) # both get here
return result
Two identical requests arriving 5ms apart both pass the if. Let the database enforce it instead:
# ✅ The unique constraint does the real work
try:
with db.transaction():
record = db.insert_idempotency_record(key, status="in_progress")
except UniqueConstraintViolation:
return await_or_return_existing(key)
result = charge_customer(amount)
db.update(record, status="complete", result=result)
return result
The parts that actually take work:
- Idempotency has to be atomic with the operation itself. If you record the key in one transaction and charge in another, you've just moved the race condition somewhere less obvious.
- Concurrent duplicates are the hard case. A naive "check then write" won't catch them. You need a unique constraint or a lock doing the real work, not application logic hoping for the best.
- Keys need a sensible lifetime. Too short and legitimate retries slip through. Too long and you're storing a permanent record of every request you've ever seen.
- It has to extend past your API boundary. Your internal queues, your workers, your webhook handlers, every hop is another opportunity to duplicate. Idempotency at the edge with at-least-once processing behind it just moves the problem inward.
Idempotency is really just the discipline of making your system's behavior depend on intent rather than on delivery, and delivery is the thing you don't control.
2. ⚠️ Every gateway you integrate is a vendor, and vendors fail
When you integrate a payment gateway, you're not adding a library. You're taking on a dependency whose code you can't read, whose deploys you don't schedule, whose incidents you learn about from Twitter, and whose SLA is a document, not a guarantee.
I stopped asking "what if this provider goes down" a long time ago. The right question is "what do we do when this provider goes down," and it's a question with a schedule attached, because it will happen this quarter.
That shift in framing changes the architecture:
🧱 You isolate them
Every provider sits behind your own interface, speaking your own domain language. Your core system should never know that Provider A calls it a transaction_reference and Provider B calls it a paymentId.
┌──────────────────────┐
│ Payment Service │ ← speaks YOUR domain language
└──────────┬───────────┘
│
┌──────────▼───────────┐
│ Provider Interface │ ← translation lives here
└──┬────────┬────────┬─┘
│ │ │
┌───▼──┐ ┌───▼──┐ ┌───▼──┐
│ A │ │ B │ │ C │ ← vendors. they will fail.
└──────┘ └──────┘ └──────┘
That translation layer is annoying to build, and it is the thing that lets you swap a provider under load instead of during a two-month migration.
⏱️ You assume timeouts, not errors
A clean error is a gift, it tells you what happened. The genuinely dangerous response is no response.
Your request may have been processed. It may not have been. You don't know, and you can't just retry blindly. This is where reconciliation and status polling stop being nice-to-haves: when the network gives you ambiguity, you need a way to go ask "did this actually happen?" and get an authoritative answer.
🔌 You add circuit breakers
When a provider starts failing, hammering it with retries makes their recovery slower and your queues longer. Fail fast, back off, route elsewhere, come back later.
🎲 You retry with jitter, and only where it's safe
delay = min(base * (2 ** attempt), max_delay)
delay = random.uniform(0, delay) # ← the line people skip
Exponential backoff without jitter just means all your retries collide again in a synchronized wave. And "safe to retry" is only true because you did the work in lesson one.
🔀 You route around damage
If you're on multiple providers, the point isn't just pricing leverage, it's that you can shift volume when one degrades.
But automatic failover is only trustworthy if you can tell the difference between "provider is down" and "provider is correctly declining these transactions." Failing over on legitimate declines is a great way to turn a small problem into a fraud incident.
🧪 You never trust the sandbox
Test environments are clean, fast, and always available. Production is none of those things. The behaviors that hurt you, partial failures, delayed settlement, out-of-order webhooks, undocumented status codes, almost never show up until real traffic does.
3. 📡 You should never learn about an outage from a customer
If a customer is telling you something is broken, you've already lost twice: once for the failure, and once for not knowing about it first.
Monitoring in payments has to work on two levels, and most teams only build one.
| Level | What you measure | What it tells you |
|---|---|---|
| Technical | Latency, error rates, queue depth, saturation, provider response times, retry counts | The system is unhealthy |
| Business | Success rate by provider / channel / card type / country, volume vs. last week, transactions stuck pending, settlement vs. ledger | Money is not moving correctly |
The reason you need both is that the worst payments incidents don't look like outages.
Every service is up. Latency is fine. Error rates are flat. And a specific bank's cards have been silently failing for forty minutes because a provider quietly changed something.
Infrastructure metrics will never catch that. A success-rate alert segmented by issuer catches it in five minutes.
A few things I now consider non-negotiable:
- Alert on trends, not just thresholds. A drop from 94% to 71% success is an emergency even though 71% isn't zero.
- Every alert needs a runbook. An alert that fires at 3am with no attached "here's what to check and who to call" is just anxiety with a pager.
- Traceability end to end. When someone asks about one specific transaction, you should be able to reconstruct its entire life, every state change, every provider call, every retry, without SSH-ing into anything.
- Ops needs their own tools. Your support and operations teams shouldn't be filing engineering tickets to answer routine customer questions. Give them read access to transaction state and safe, audited actions. This buys back an astonishing amount of engineering time.
Availability is the other half of this. In most products, downtime costs you some engagement. In payments, downtime is transactions that didn't happen, revenue that evaporates and doesn't come back, plus something more expensive: trust that erodes a little each time and doesn't rebuild at the same rate. People will forgive a slow app. They get quietly nervous about a payment system that failed on them once, and they stop reaching for it first.
4. 🧾 The things nobody warns you about
A few more lessons that cost me something to learn.
Your ledger is the source of truth, not your provider's dashboard
Build a proper double-entry, append-only ledger early. Never update a balance in place. Never delete an entry. Corrections are new entries.
-- ❌ Where did the money go? Nobody knows.
UPDATE accounts SET balance = balance - 500 WHERE id = 42;
-- ✅ Append-only. The history IS the balance.
INSERT INTO ledger_entries (account_id, amount_minor, currency, direction, ref)
VALUES (42, 50000, 'KES', 'debit', 'txn_9f2a'),
(17, 50000, 'KES', 'credit', 'txn_9f2a');
When something eventually goes wrong, and it will, the only thing that saves you is an immutable record of what your system believed at every point in time.
Reconcile continuously, not monthly
Automated comparison between your ledger and each provider's settlement reports, running daily at minimum. Discrepancies compound. A mismatch found the next morning is a fix; the same mismatch found at quarter-end is an investigation.
Never use floating point for money
0.1 + 0.2 === 0.3 // false. this is your revenue.
Store minor units as integers. Always attach a currency. Decide your rounding rules explicitly and write them down, because rounding disagreements between you and a provider will eventually surface as a real, unexplainable gap.
Model payment state as an explicit state machine
Not a boolean. Not a status string that anyone can set to anything.
initiated ──► pending ──┬─► succeeded ──► refunded
│
├─► failed
│
└─► expired
# succeeded ──► pending is NOT a valid transition.
# Enforce that in ONE place, not in seven services.
Half the weird bugs in payments are illegal state transitions that nobody thought to prevent.
Webhooks arrive out of order, late, or twice
Sometimes the "success" event lands before the "processing" event. Handle each one as a fact about a point in time, not as an instruction to advance a status.
Design the unhappy paths first
Refunds, partial refunds, reversals, chargebacks, disputes, expired authorizations. Teams build the happy path, ship it, and then discover that refunds don't fit the data model at all. The unhappy paths are where the actual complexity lives, and retrofitting them is far more expensive than designing for them.
Compliance is an architectural constraint, not a checklist
What you're allowed to store, where you're allowed to store it, who can see it, how long you keep it. Discovering these requirements after you've built is a rewrite.
5. ⚖️ Everything above is in tension with everything else
Here's the part that took me longest to accept: none of these lessons is free, and several of them actively fight each other.
| What you gain | What it costs you |
|---|---|
| Aggressive retries → higher success rates | More duplicate risk |
| Multi-provider redundancy → resilience | Integration surface, reconciliation complexity, on-call load |
| Comprehensive monitoring → catch real problems | Alert fatigue that slows your real response |
| Meticulous ledger → certainty | Write throughput |
There's no configuration where you max out every dial. The engineering is in deciding, deliberately and with your eyes open, where you sit on each of these, and being honest that it's a choice with a cost, rather than pretending you got everything.
🎯 If you're starting on this today
Build for the failure cases before you build for scale. Scale problems announce themselves loudly and you get to fix them in daylight. Correctness problems hide, compound quietly, and surface at the worst possible time, usually as a number that doesn't add up and a customer who wants to know where their money went.
Get the boring parts right:
- [ ] Idempotency keys, enforced at the database level
- [ ] An append-only, double-entry ledger
- [ ] Monitoring on business metrics, not just infrastructure
- [ ] An explicit state machine for payment status
- [ ] Automated daily reconciliation
- [ ] Refunds and chargebacks in the data model from day one
Everything else is much easier to build on top of that than it is to bolt on afterward.
Have you hit any of these the hard way? I'd like to hear which one bit you, the comments are open. 👇
Top comments (0)