Every industry has different constraints that shape its infrastructure decisions. Banking cares about compliance and audit trails. E-commerce cares about traffic spikes on sale days. Fintech payment systems care about something more specific: money cannot be created or destroyed by accident.
If a request gets retried because of a flaky network connection, a customer should never be charged twice. If two requests hit the database at the same time, the ledger should never end up in an inconsistent state. This one constraint, correctness under concurrency and failure, ends up driving almost every infrastructure decision in a payment system.
I wanted to understand this properly, so I built a small payment gateway simulation called ledger-sentinel to work through the actual engineering problems, not just read about them. Here's what the DevOps/infrastructure thinking behind a system like this looks like, using what I built as a concrete example.
The core problem: idempotency
The first design question in any payment system is: what happens when the same request arrives twice?
This happens constantly in production; a client times out and retries, a load balancer resends a request, a mobile app has a flaky connection and fires the same "pay now" tap twice. If your system isn't built to handle this, you get duplicate charges.
The standard pattern is an idempotency key: the client sends a unique key with each logical transaction, and the server guarantees that request only actually processes once, no matter how many times it arrives.
In ledger-sentinel, I implemented this with:
A Redis distributed lock (SET NX EX) that acts as a fast, short-lived gatekeeper... the first request to arrive with a given key gets the lock, any duplicate arriving while it's still processing gets rejected (HTTP 409) instead of double-processing
A response cache layered on top, so once a request has actually completed, any later retry with the same key gets served the original cached response (HTTP 200) instead of hitting the database again
This is the difference between "the system happened to work in testing" and "the system is provably safe under concurrent retries", and it's the first thing I'd want to verify actually holds before trusting any payment infrastructure.
Why async settlement, not synchronous processing
A naive payment API might try to validate, charge, and update the ledger all within a single HTTP request. This is a mistake at any real scale, because it means the client is waiting on your slowest downstream dependency (usually the database) for every single request, and a slow ledger write becomes a slow, unreliable API.
The fix is decoupling ingestion from settlement:
The API validates the request and idempotency key, then pushes the transaction onto a RabbitMQ queue
A separate worker process consumes the queue and does the actual ledger update
The client gets a fast response (validated and queued), and the ledger settles asynchronously
This also gives you backpressure control, if the worker is falling behind under a traffic burst, messages queue up safely instead of the whole system falling over. I set a consumer prefetch limit (prefetch_count=10) specifically so the worker never grabs more in-flight work than it can actually handle at once.
Why the ledger itself still needs strict locking
Queueing solves the "don't block the client" problem, but it doesn't solve the "two workers processing related transactions at the same time" problem. For that, the actual database work still needs strict consistency guarantees.
I used PostgreSQL row-level locking (SELECT ... FOR UPDATE) so that when a worker is updating a specific ledger balance, no other transaction can read or write that same row until the lock releases. This is slower than an unlocked read, but for a ledger, "slower but always correct" beats "fast but occasionally wrong", the entire point of a ledger is that the numbers are always right.
How I actually verified this wasn't just "should work in theory"
Design decisions are cheap to write down and easy to get wrong in practice. So I stress-tested it with k6, firing over 1,000 concurrent transaction requests at the system, some with duplicate idempotency keys on purpose, and checked:
Zero duplicate charges — verified directly against the database (GROUP BY idempotency_key HAVING COUNT(*) > 1 returned nothing)
p99 latency stayed sub-second even during the burst, and recovered to ~4–8ms once the burst normalized and requests were hitting the cache
100% success rate, zero HTTP 5xx errors, across the full load test
I also watched this fail in interesting ways during development, for example, my first version of the load test accidentally reused the same idempotency key across requests, which meant Redis was serving cached responses without ever reaching RabbitMQ at all. The queue graphs looked completely flat, which looked like a bug until I realized the test script itself was wrong, not the system. That kind of debugging, figuring out whether the system or the test is lying to you, is most of what this work actually is.
What I'd add next if this were a real production system
This project is a simulation, not a production system, and there are things a real fintech payment platform would need that I haven't built here:
Multi-region failover, since payment infra can't have a single point of failure
Real reconciliation against an external payment processor's records, not just internal consistency
Proper secrets management and PCI-DSS-aligned access controls, rather than local .env files
Chaos testing (deliberately killing a worker mid-transaction) rather than just load testing
But the core lesson holds even at small scale: in payment systems, the interesting infrastructure decisions aren't about which cloud provider or which orchestrator you pick. They're about how you handle the moment two things happen at exactly the same time.
Top comments (0)