What Would a Fintech Payment System's DevOps Setup Actually Look Like?

Okikiola Ashiru — Wed, 29 Jul 2026 14:10:43 +0000

Every industry has different constraints that shape its infrastructure decisions. Banking cares about compliance and audit trails. E-commerce cares about traffic spikes on sale days. Fintech payment systems care about something more specific: money cannot be created or destroyed by accident.

If a request gets retried because of a flaky network connection, a customer should never be charged twice. If two requests hit the database at the same time, the ledger should never end up in an inconsistent state. This one constraint, correctness under concurrency and failure, ends up driving almost every infrastructure decision in a payment system.

I wanted to understand this properly, so I built a small payment gateway simulation called ledger-sentinel to work through the actual engineering problems, not just read about them. Here's what the DevOps/infrastructure thinking behind a system like this looks like, using what I built as a concrete example.

The core problem: idempotency

The first design question in any payment system is: what happens when the same request arrives twice?

This happens constantly in production; a client times out and retries, a load balancer resends a request, a mobile app has a flaky connection and fires the same "pay now" tap twice. If your system isn't built to handle this, you get duplicate charges.

The standard pattern is an idempotency key: the client sends a unique key with each logical transaction, and the server guarantees that request only actually processes once, no matter how many times it arrives.

In ledger-sentinel, I implemented this with:

A Redis distributed lock (SET NX EX) that acts as a fast, short-lived gatekeeper... the first request to arrive with a given key gets the lock, any duplicate arriving while it's still processing gets rejected (HTTP 409) instead of double-processing
A response cache layered on top, so once a request has actually completed, any later retry with the same key gets served the original cached response (HTTP 200) instead of hitting the database again

This is the difference between "the system happened to work in testing" and "the system is provably safe under concurrent retries", and it's the first thing I'd want to verify actually holds before trusting any payment infrastructure.

Why async settlement, not synchronous processing

A naive payment API might try to validate, charge, and update the ledger all within a single HTTP request. This is a mistake at any real scale, because it means the client is waiting on your slowest downstream dependency (usually the database) for every single request, and a slow ledger write becomes a slow, unreliable API.

The fix is decoupling ingestion from settlement:

The API validates the request and idempotency key, then pushes the transaction onto a RabbitMQ queue
A separate worker process consumes the queue and does the actual ledger update
The client gets a fast response (validated and queued), and the ledger settles asynchronously

This also gives you backpressure control, if the worker is falling behind under a traffic burst, messages queue up safely instead of the whole system falling over. I set a consumer prefetch limit (prefetch_count=10) specifically so the worker never grabs more in-flight work than it can actually handle at once.

Why the ledger itself still needs strict locking

Queueing solves the "don't block the client" problem, but it doesn't solve the "two workers processing related transactions at the same time" problem. For that, the actual database work still needs strict consistency guarantees.

I used PostgreSQL row-level locking (SELECT ... FOR UPDATE) so that when a worker is updating a specific ledger balance, no other transaction can read or write that same row until the lock releases. This is slower than an unlocked read, but for a ledger, "slower but always correct" beats "fast but occasionally wrong", the entire point of a ledger is that the numbers are always right.

How I actually verified this wasn't just "should work in theory"

Design decisions are cheap to write down and easy to get wrong in practice. So I stress-tested it with k6, firing over 1,000 concurrent transaction requests at the system, some with duplicate idempotency keys on purpose, and checked:

Zero duplicate charges — verified directly against the database (GROUP BY idempotency_key HAVING COUNT(*) > 1 returned nothing)
p99 latency stayed sub-second even during the burst, and recovered to ~4–8ms once the burst normalized and requests were hitting the cache
100% success rate, zero HTTP 5xx errors, across the full load test

I also watched this fail in interesting ways during development, for example, my first version of the load test accidentally reused the same idempotency key across requests, which meant Redis was serving cached responses without ever reaching RabbitMQ at all. The queue graphs looked completely flat, which looked like a bug until I realized the test script itself was wrong, not the system. That kind of debugging, figuring out whether the system or the test is lying to you, is most of what this work actually is.

What I'd add next if this were a real production system

This project is a simulation, not a production system, and there are things a real fintech payment platform would need that I haven't built here:

Multi-region failover, since payment infra can't have a single point of failure
Real reconciliation against an external payment processor's records, not just internal consistency
Proper secrets management and PCI-DSS-aligned access controls, rather than local .env files
Chaos testing (deliberately killing a worker mid-transaction) rather than just load testing

But the core lesson holds even at small scale: in payment systems, the interesting infrastructure decisions aren't about which cloud provider or which orchestrator you pick. They're about how you handle the moment two things happen at exactly the same time.

Repo: https://github.com/aashiruu/ledger-sentinel

Hi, I'm Okikiola. Cloud & DevOps Engineer working through infrastructure problems in public

Okikiola Ashiru — Wed, 29 Jul 2026 10:05:55 +0000

Hi, I'm Okikiola — a Cloud & DevOps Engineer based in Lagos, Nigeria.

I came into this field a little sideways. My background is in Biochemistry, but somewhere along the way I got pulled toward infrastructure... the kind of problems where something breaks under load and you have to figure out why, or where a system needs to survive a traffic burst without falling over or double-charging someone.

That pull turned into two years of hands-on work: provisioning cloud environments on AWS and OCI, building CI/CD pipelines, and setting up the observability stacks that catch failures before a customer notices them.

What I work with day to day

Infrastructure as Code: Modular Terraform across AWS and OCI, managing VPCs, IAM policies, and multi-node EKS clusters across projects.
CI/CD: GitHub Actions workflows for automated linting, container builds, and deployment.
Observability: Prometheus, Grafana, and Loki for metrics, logs, and alerting.
Automation: Go and Python (Boto3) scripts for the unglamorous but necessary work, cleaning up orphaned cloud resources, enforcing tagging policies, automating backup rotations.

Why I'm writing here

I've started building deeper projects specifically to understand infrastructure problems at the system level, not just to add tools to a resume.

My most recent one, ledger-sentinel, is a simulated fintech payment gateway I built to work through idempotency locking (Redis SET NX EX), async settlement via RabbitMQ, and database consistency under concurrent load. It's the kind of system where "it worked in my test" and "it's actually safe against double-charges" are two very different claims, and I wanted to prove the second one, not just assume it.

I plan to write about this project and others like it, real architecture decisions, load-test numbers, what broke, and what I'd still need to add before calling any of it production-ready. I'd rather be upfront about the gap between a personal project and a production system than pretend it isn't there.

What I'm currently digging into

Right now I'm spending time on distributed lock contention patterns, Redis cache short-circuiting, and queue backpressure mechanics, mostly because ledger-sentinel showed me how much nuance sits behind "just add a distributed lock" once thousands of requests hit the same endpoint at once.

If you work in DevOps, SRE, or platform engineering, or you're navigating a non-traditional path into cloud infrastructure yourself, I'd like to connect. Always glad to talk through Terraform patterns, observability setups, or a race condition you've had to hunt down.

GitHub: github.com/aashiruu
LinkedIn: linkedin.com/in/okikiolaashiru