I Built a Scalable Financial Transaction System That Stays Correct Under Load

Praval Parikh — Sat, 03 Jan 2026 13:28:12 +0000

Financial systems rarely fail because they are slow.

They fail because they are wrong.

A duplicated transaction, an out-of-order withdrawal, or a race condition in a balance update is enough to permanently break user trust. As traffic grows, correctness becomes harder not easier.

In this post, I’ll walk through how I designed a scalable financial transaction system that prioritizes correctness first, while still achieving high throughput under heavy concurrency.

👉 Source code:

https://github.com/plebsicle/microfin

Why Naive Financial Systems Break at Scale

A typical starting point looks simple:

A request hits the server
The server validates it
A database transaction updates the balance

This works fine at low traffic. Under concurrency, it quickly breaks.

Common failure modes:

Multiple concurrent updates to the same balance
Database connection exhaustion
Lock contention increasing latency
Race conditions causing incorrect balances

Even with database transactions, allowing concurrent writes to shared financial state is dangerous. Locking harder reduces throughput, and scaling vertically only delays the problem.

Direct concurrent writes to financial state do not scale safely.

Defining the Non-Negotiables

Before choosing any technology, I defined the system requirements:

Strong consistency for account balances
Sequential execution of transactions per user/account
High throughput under concurrent load
Horizontal scalability
Fault tolerance
A system that is easy to reason about

Every architectural decision follows from these constraints.

High-Level Architecture

The key idea is to separate request ingestion from state mutation.

At a high level:

Requests are accepted and validated by application servers
Valid transactions are queued
Transactions are processed sequentially per user
Durable state is updated in the database
Caching is used purely as a performance optimization

This allows the system to absorb traffic spikes without immediately mutating shared state.

System Architecture Diagram

The diagram below shows the high-level architecture of the system and how requests flow through different components, from request ingestion to durable storage.

Each incoming request is first load-balanced by Nginx, validated by application servers, and then published to Kafka. Kafka enforces strict per-user ordering before workers update PostgreSQL (the system of record) and Redis (the cache).

The Core Insight: Ordering Beats Locking

The most important insight in this design is simple:

Correct ordering is more scalable than locking.

Instead of allowing concurrent updates and then trying to protect state with locks, the system ensures that transactions for a given user are never processed concurrently.

This is achieved through deterministic ordering, not mutual exclusion.

Enforcing Sequential Execution with Kafka Partitioning

Each transaction is associated with a user ID / account number.

A hash of this identifier determines which Kafka partition the transaction is sent to.

This gives us three critical guarantees:

All transactions for a user always go to the same partition
Each partition is consumed by exactly one worker
Kafka guarantees ordering within a partition

As a result:

No two transactions for the same user are processed concurrently
Race conditions are eliminated
Database-level locking is unnecessary

Scaling is achieved by increasing the number of partitions and workers, while preserving per-user ordering.

What Happens When a User Double-Clicks?

Users double-click buttons, retry requests, or refresh pages during slow network conditions.

In this system:

Each request is treated as a distinct transaction
Multiple similar requests may be enqueued
They are processed strictly sequentially

Rather than rejecting or deduplicating requests at the application layer, the system relies on deterministic execution order to guarantee correctness.

Why Kafka

Kafka isn’t used here just as a queue, it’s a core correctness mechanism.

Kafka provides:

Strong ordering guarantees within partitions
Horizontal scalability through partitioning
Fault tolerance via replication

Kafka producers are configured with idempotency enabled. This ensures that:

Duplicate messages are not written during retries
Ordering is preserved
Exactly-once production semantics are achieved per producer session

This is producer-side idempotency, not consumer-side deduplication. Correctness comes from ordering, not from rejecting duplicate-looking requests.

PostgreSQL as the Source of Truth

PostgreSQL acts as the system of record.

Why PostgreSQL:

Strong ACID guarantees
Mature transactional behavior
Predictable correctness under concurrency

PgBouncer is used for connection pooling to prevent database exhaustion under heavy load.

All balance updates occur inside database transactions, ensuring durability and correctness even in failure scenarios.

Redis: Performance Optimization, Not a Dependency

Redis is used for:

Session management
Caching frequently accessed data
Reducing read load on PostgreSQL

Redis is not part of the correctness path. There is no distributed transaction protocol between PostgreSQL and Redis. If Redis fails, the system remains correct — only performance is affected.

End-to-End Transaction Flow

Client sends a transaction request
Nginx load-balances the request
Application server validates it
Transaction is published to Kafka
Kafka assigns it to a partition via hashing
A worker processes transactions sequentially
PostgreSQL updates occur inside a transaction
Redis is updated opportunistically
Response is returned to the client

Load Testing and Results

The system was stress-tested using K6 with scenarios including:

Account creation
User sign-ins
Deposits, withdrawals, and transfers

Results:

Sustained 3600+ requests per second
100% success rate
Sub-second p95 latencies

Correctness was maintained throughout.

Observability (and Its Limits)

Prometheus and Grafana are used to monitor:

Request rate
Latency
CPU and memory usage of application servers

Infrastructure-level metrics for Kafka, Redis, and PostgreSQL are not instrumented in this version.

Trade-offs and Limitations

This system intentionally avoids:

Semantic request deduplication
Distributed transactions across PostgreSQL and Redis
Full infrastructure-level observability

These trade-offs keep the system simpler and more reliable.

What I’d Improve Next

Future improvements could include:

Kafka, Redis, and PostgreSQL metrics
Database sharding strategies
Advanced failure recovery mechanisms

Final Takeaways

Ordering simplifies concurrency
Correctness matters more than raw speed
Distributed systems are about trade-offs, not perfection
Designing for correctness early prevents painful rewrites

Scalability isn’t about handling more requests —

it’s about handling them correctly.

🔗 Project Link

https://github.com/plebsicle/microfin

DEV Community: Praval Parikh