Imagine this:
It's salary day.
It's 2 PM.
Millions of people across India suddenly open their UPI apps and start paying rent, sending money to family, paying credit-card bills, and shopping online.
Now here's the system-design interview question:
If millions of people make payments at almost exactly the same time, is every request hitting one central server? What prevents the entire payment system from freezing?
At first glance, it sounds like a scaling problem.
It isn't just a scaling problem.
It's a combination of:
- horizontal scaling
- concurrency
- distributed systems
- database consistency
- retries
- idempotency
- backpressure
- failure isolation
- downstream bottlenecks
And that's what makes payment systems such an interesting system-design problem.
First: Don't Imagine One Giant UPI Server
A common mental model looks like this:
Millions of users
|
v
+-------------+
| UPI Server |
+-------------+
|
v
Bank
If that were literally true, we'd have a pretty serious problem.
One machine cannot safely process the country's entire payment traffic.
Instead, think about a distributed system:
Users
|
v
+---------------+
| API / Gateway |
+---------------+
/ | \
/ | \
v v v
[S1] [S2] [S3]
| | |
+------+------+
|
Payment Services
|
+--------+--------+
| |
Bank A Bank B
The exact implementation of a real payment network is much more complicated than this diagram, but this is the right system-design mental model.
The important idea is:
The system is distributed across many machines and participating institutions.
Step 1: The First Problem — Traffic Spikes
Let's take a concrete example.
You want to pay your landlord:
₹25,000
At the same moment, millions of other people are doing something similar.
Suddenly:
Normal traffic:
100K requests/sec
Salary day:
████████████████████████
1M+ requests/sec
The first question is:
How do we handle the additional traffic?
Naive Solution: One Powerful Server
We could buy a massive machine.
1M requests/sec
|
v
+---------------+
| HUGE SERVER |
| 256 CPU cores |
| 2 TB RAM |
+---------------+
This is called vertical scaling.
Make the machine bigger.
But there are limits.
Eventually:
CPU → limit
Memory → limit
Network → limit
Connections → limit
And there's an even bigger problem.
If the server dies:
💥
|
v
+-------------+
| One Server |
+-------------+
|
v
Entire system
DOWN
For a payment system, that's unacceptable.
Solution: Horizontal Scaling
Instead of making one machine enormous, add more machines.
Requests
|
v
+---------------+
| Load Balancer |
+---------------+
/ | \
/ | \
v v v
[S1] [S2] [S3]
| | |
[S4] [S5] [S6]
| | |
... ... ...
If one server handles 20,000 requests/sec, and we need roughly 1 million:
1,000,000 / 20,000 ≈ 50 servers
We can scale the application tier horizontally.
Now if one server fails:
S1 💥
S2
S3
S4
S5
...
Traffic can be routed to healthy servers.
This gives us our first reusable system-design pattern:
Pattern #1: Horizontal Scaling
When request volume exceeds the capacity of one machine, distribute requests across many machines.
But We've Created a New Problem
Suppose two of your payments arrive simultaneously.
Payment A:
Harsh → Landlord ₹25,000
Payment B:
Harsh → Amazon ₹20,000
They might land on different servers:
Load Balancer
/ \
v v
[S1] [S2]
| |
+------?------+
|
Bank
Suppose Harsh has:
₹30,000
Both servers read the balance at approximately the same time.
Server 1 sees:
₹30,000
Server 2 sees:
₹30,000
Then:
S1:
₹30,000 - ₹25,000
= ₹5,000
S2:
₹30,000 - ₹20,000
= ₹10,000
The system has effectively allowed:
₹45,000
to be spent from an account containing:
₹30,000
That's a race condition.
And this is where payment-system design gets interesting.
Step 2: We Need Atomic State Changes
A payment isn't just:
UPDATE balance
Conceptually, we need something closer to:
Check balance
↓
Verify payment
↓
Debit sender
↓
Credit receiver
↓
Record transaction
The critical state transition must happen safely under concurrency.
We need a guarantee that two competing operations cannot both incorrectly modify the same financial state.
Depending on the architecture, this can involve:
- database transactions
- locking
- optimistic concurrency control
- serialization
- partition ownership
- carefully designed state machines
The important interview lesson isn't:
"Use database locks."
It's:
Identify the shared mutable state and protect the critical transition.
Step 3: Now the Database Becomes the Bottleneck
Let's say we've successfully scaled our application servers.
We now have:
Load Balancer
|
+--------------+--------------+
| | |
S1 S2 S3
| | |
+--------------+--------------+
|
v
+-----------+
| Database |
+-----------+
We have 500 application servers.
But one database.
Now all those servers are fighting for the same resource.
500 servers
|
|
v
+-----------+
| DB |
| 💥 |
+-----------+
The application tier scales.
The database doesn't.
This is a classic distributed-systems bottleneck:
The fastest part of your system doesn't matter if a slower shared dependency limits the entire system.
Step 4: Partition the Work
Instead of forcing everything through one database/resource, we can partition data and workload.
Conceptually:
Payment Requests
|
+-------+-------+
| |
v v
Partition A Partition B
| |
DB-A DB-B
The partitioning strategy could be based on something like:
account ID
bank
customer ID
transaction domain
geographical region
The exact choice depends on the system.
This is generally called:
Sharding / Partitioning
The reusable pattern is:
Pattern #2: Partition the bottleneck
When one resource can't handle the workload, divide the workload into independent partitions.
But There's Another Problem
Imagine our application servers are perfectly healthy.
Our databases are perfectly healthy.
But a downstream bank suddenly becomes overloaded.
Payment Services
/ | \
v v v
Bank A Bank B Bank C
|
v
💥
Our system can process millions of incoming requests.
But the downstream dependency might only safely process a smaller amount.
This gives us another fundamental principle:
The capacity of a distributed system is constrained by its critical bottlenecks and dependencies.
You can't solve a downstream bottleneck by simply adding more application servers.
Step 5: Backpressure
Suppose a component can safely process:
100K operations/sec
but we're receiving:
300K operations/sec
If we blindly forward everything:
300K
|
v
[Processor]
|
💥
Instead, for work that is safe to process asynchronously, we can introduce a buffer:
300K requests
|
v
+-------+
| Queue |
+-------+
|
| 100K/sec
v
+-----------+
| Processor |
+-----------+
The queue absorbs temporary bursts.
This gives us:
Pattern #3: Backpressure
When producers can generate work faster than consumers can process it, slow down producers or buffer the work.
This is where technologies such as Kafka or other messaging systems become useful.
But notice the reasoning.
We didn't start with:
"Let's use Kafka."
We started with:
"Our producer is faster than our consumer. We need buffering/backpressure."
Then we choose a technology.
Can We Queue the Entire Payment?
Not necessarily.
This is an important interview trap.
You shouldn't say:
"We'll just put every payment into Kafka."
Financial transactions have correctness and latency requirements.
There is a difference between:
Critical payment state transition
and:
Side effects
For example:
Payment
|
+----> Update financial state
|
+----> Send notification
|
+----> Update analytics
|
+----> Generate receipt
|
+----> Update recommendation system
The financial state transition may require strict correctness.
But sending:
"₹25,000 paid successfully"
to a notification service doesn't necessarily need to block the core transaction.
Those side effects can often be asynchronous.
Step 6: Now Imagine the Network Fails
Here's where payment systems become really interesting.
You press:
PAY ₹25,000
The request reaches the payment system.
The money gets debited.
But before the response reaches your phone:
Bank → 💥 network timeout 💥 → Phone
Your app says:
Payment failed
You think:
"Okay, I'll try again."
So you press Pay again.
Now the system receives:
Payment #1
₹25,000
Payment #2
₹25,000
If the system treats both as new payments:
Your account
|
+-- ₹25,000
|
+-- ₹25,000
|
v
₹50,000 debited
That's obviously unacceptable.
Step 7: Idempotency
We need the system to recognize:
"This retry is actually the same payment."
So the client/request can carry a unique identifier:
payment_id = ABC123
First request:
ABC123 → process payment
Retry:
ABC123 → already processed
The system returns the previous result rather than charging the user again.
Conceptually:
Request
|
v
+----------------------+
| payment_id = ABC123 |
+----------------------+
|
v
Already exists?
/ \
YES NO
| |
v v
Return result Process
This is idempotency.
And it is one of the most important concepts in distributed systems.
Pattern #4: Idempotency
A retry of the same logical operation should not produce an additional side effect.
You will see this pattern everywhere:
- payments
- order creation
- payment webhooks
- message processing
- API retries
- distributed jobs
Step 8: Why Retries Are Dangerous
Retries sound harmless:
Request failed
↓
Retry
↓
Retry
↓
Retry
But imagine 1 million clients doing this.
1M requests
|
failure
|
+-----+-----+
| |
retry retry
| |
+-----+-----+
|
retry
|
v
💥💥💥
A failing system can actually make itself more overloaded through retries.
This is called a retry storm.
So production systems need things like:
- exponential backoff
- jitter
- retry limits
- timeouts
- circuit breakers
- idempotency
Again, each mechanism exists because a specific failure mode exists.
The Bigger Picture
At this point our system is evolving.
We started with:
User
|
v
Server
|
v
Bank
And progressively discovered problems.
Traffic spike
Load Balancer
/ | \
S1 S2 S3
Shared state
Multiple servers
|
v
Concurrency control
Database bottleneck
|
partition
/ \
DB-A DB-B
Downstream overload
Producer
|
Queue / Backpressure
|
Consumer
Network failure
Request
|
timeout
|
retry
Retry duplication
payment_id
|
idempotency
Our architecture is now becoming:
Users
|
v
+-------------+
| Gateway |
+-------------+
|
Load balancing
|
+-------------+-------------+
| | |
S1 S2 S3
| | |
+-------------+-------------+
|
Payment Service
|
+------+------+
| |
Partition A Partition B
| |
DB-A DB-B
|
Critical transaction
|
+-----+------+
| |
Async work Notifications
|
Queue
And around the entire system:
timeouts
retries
idempotency
rate limits
backpressure
monitoring
failure isolation
The Most Important Lesson
When you're asked:
"Design a UPI-scale payment system."
Don't start drawing 20 boxes.
Start asking:
1. What's the workload?
Requests/sec?
Peak requests/sec?
Average transaction size?
Read/write ratio?
2. What's the critical state?
Account balance
Transaction status
Payment ID
3. Where is concurrency dangerous?
Two payments touching the same account
4. What's the bottleneck?
Application?
Database?
Network?
Downstream bank?
5. What happens during failure?
Timeout
Crash
Duplicate request
Partial success
Network partition
6. What happens when traffic suddenly increases?
Scale
Buffer
Rate limit
Backpressure
7. What happens when the client retries?
Idempotency
This is the real system-design thought process.
A Reusable Pattern Library
By solving this one problem, we've already learned several patterns.
| Problem | Pattern |
|---|---|
| One server can't handle traffic | Horizontal scaling |
| Shared state modified concurrently | Atomicity / concurrency control |
| One DB becomes bottleneck | Sharding / partitioning |
| Producer faster than consumer | Queue / backpressure |
| Request repeated after timeout | Idempotency |
| Dependency becomes slow/unavailable | Timeout / circuit breaker |
| Huge traffic spike | Rate limiting / load shedding |
| Temporary downstream failure | Retry + backoff |
And this is exactly how you should prepare for SDE-2 system design.
Don't memorize architectures.
Memorize the problem → failure → pattern relationship.
One Final Mental Model
When you see:
"Millions of requests arrive simultaneously."
Think:
HUGE TRAFFIC
|
v
Can one machine?
/ \
NO YES
|
v
Horizontal scaling
|
v
What's shared?
|
v
Concurrency problem?
|
v
Protect state transition
|
v
What's the bottleneck?
|
v
Partition / scale
|
v
What if dependency slows?
|
v
Backpressure / queue
|
v
What if request retries?
|
v
Idempotency
|
v
What if it fails?
|
v
Timeout / retry / recovery
That's the mental model I'd want an SDE-2 candidate to have.
The interesting part of this problem isn't "How does UPI handle 1,000+ crore transactions?"
It's:
How do you build a distributed system where massive concurrency, partial failures, retries, and bottlenecks don't turn a single ₹25,000 payment into a financial disaster?
And once you understand that, you can reuse the same reasoning for Cloudflare rate limiting, ticket booking, food delivery, Uber, payment gateways, distributed job systems, and almost every large-scale backend system.
Top comments (0)