System Design from Developer to Architect --- Part 1
In Part 0, we started with a simple rule: don't add architecture
because it looks scalable. Add it when a real problem makes it
necessary.In this part, we'll start with one user, keep adding traffic, and
change the architecture only when something gives us a reason to.
Previous: Part 0 --- How to Think About System Design Without Just
Drawing
Boxes
A million users sounds like a completely different engineering problem
from one user.
And eventually, it is.
But the interesting part isn't the final architecture.
It's how we get there.
We won't begin with CDN, Redis, replicas, event brokers, and a cluster
of services.
We'll begin here:
User → Application → Database
Then we'll keep increasing traffic.
Every time something becomes a real constraint, we'll ask:
What broke, why did it break, and what is the smallest architectural
change that solves it?
That's the scaling journey.
Stage 1 --- 1 User: Keep It Boring
Imagine we've just launched a service-booking application.
One user opens the application, searches for a professional, checks a
slot, and creates a booking.
Our architecture can be extremely simple:
User
│
▼
Application
│
▼
Database
And that's enough.
No Redis. No Kafka. No Kubernetes. No read replicas.
There is nothing wrong with this architecture.
In fact, adding distributed infrastructure now would probably make the
system harder to build, deploy, debug, and operate without solving a
real problem.
The cheapest scaling problem is the one you don't have yet.
Stage 2 --- 100 Users: The Simple Architecture Still Works
Now we have 100 users.
Do we need microservices? Probably not.
Do we need Kafka? Probably not.
At this stage, the highest-value improvements are often much less
exciting:
- Add the right database indexes
- Fix inefficient queries
- Avoid N+1 queries
- Use database connection pooling
- Compress HTTP responses
- Handle static assets efficiently
- Add basic metrics and logs
- Measure latency before optimizing it
A surprisingly large amount of scale can come from simply making the
existing system efficient.
Sometimes:
Slow API
↓
Fix query / index
↓
Fast API
is a better scaling strategy than adding another server.
Before distributing the system, make the simple system efficient.
Stage 3 --- 10,000 Users: Scale Up Before Scaling Out
Traffic keeps growing.
Now we start seeing resource pressure:
CPU 78%
Memory 82%
p95 650ms
Traffic ↑
The first response doesn't always need to be horizontal scaling.
We might simply give the machine more resources:
2 CPU → 8 CPU
4 GB → 32 GB RAM
That's vertical scaling.
It's simple because our architecture barely changes.
Vertical scaling can take us surprisingly far.
But machines don't grow forever, and a single server remains a single
failure domain.
Eventually, traffic may exceed what one instance can comfortably handle.
That's when the second server becomes interesting.
Stage 4 --- 50,000 Users: The Second Server Changes Everything
Eventually one application instance isn't enough.
So we add another.
Now we need something to decide where incoming requests should go.
Enter the load balancer.
Users
│
▼
Load Balancer
│
┌──────┴──────┐
▼ ▼
App 1 App 2
│ │
└──────┬──────┘
▼
Database
Now we can scale horizontally by adding application instances.
But this works cleanly only when those instances are interchangeable.
And that creates our next problem.
The State Problem
Suppose the user logs in through App 1.
Their session is stored in that server's memory.
Their next request reaches App 2.
Login → App 1
Session stored locally
Next request → App 2
Session = ?
The load balancer did exactly what we asked.
Our application architecture didn't.
This is why horizontal scaling often pushes us toward stateless
application servers.
For shared session state, Redis is one possible solution:
Load Balancer
│
┌────────┴────────┐
▼ ▼
App 1 App 2
│ │
└────────┬────────┘
▼
Redis
│
▼
Database
Notice the sequence:
More traffic
↓
More instances
↓
Requests move between instances
↓
Local state becomes a problem
↓
Shared/stateless state becomes useful
Redis didn't appear because Redis is fashionable.
It appeared because the architecture developed a state problem.
Stage 5 --- 100,000 Users: Repeated Reads Start to Hurt
Some requests repeatedly fetch the same information:
GET /services
GET /categories
GET /professionals/123
If that data changes infrequently, sending every request to the database
is wasteful.
Now caching has a concrete job.
Request
│
▼
Application
│
▼
Cache
/ \
HIT MISS
│ │
▼ ▼
Return DB
│
▼
Cache
│
▼
Return
A cache hit avoids an unnecessary database read.
A cache miss falls back to the source of truth.
This reduces latency and database pressure.
But caching introduces new concerns:
- TTL
- Invalidation
- Eviction
- Stale data
- Cache stampedes
- Failure behavior
For a service description, some staleness may be acceptable.
For the last available booking slot, it may not be.
Caching isn't just a performance decision. It's also a correctness
decision.
Stage 6 --- 250,000 Users: The Bottleneck Moves
The API tier looks healthy.
Then monitoring shows:
API CPU 35% ✓
API Memory 42% ✓
DB CPU 92% ⚠
DB Connections 95% ⚠
p95 Query Latency 780ms ⚠
Should we add another API server?
No.
The API isn't the bottleneck anymore.
The bottleneck moved.
App 1 ──┐
App 2 ──┤
App 3 ──┼────> DATABASE 🔥
App 4 ──┤
App 5 ──┘
A struggling database does not immediately mean:
"We need sharding."
Start with evidence:
Database pressure
│
├── Inspect slow queries
├── Verify indexes
├── Remove N+1 access
├── Reduce unnecessary reads
├── Cache appropriate data
├── Review connection usage
└── Then consider infrastructure scaling
A bad query executed across ten replicas is still a bad query.
Scale the bottleneck only after understanding the bottleneck.
Stage 7 --- 500,000 Users: Separate Reads From Writes
Suppose our workload is heavily read-oriented.
Users browse much more often than they modify data.
We may introduce read replicas:
Application
│
┌──────────┴──────────┐
▼ ▼
Writes Reads
│ │
▼ ▼
Primary Read Replicas
│ │
▼ ▼
Replica Replica
Writes continue to go to the primary.
Read-heavy traffic can be distributed across replicas.
But we've bought a new problem:
Replication Lag
Replication is not always instantaneous.
A user may create a booking on the primary and immediately read from a
replica that hasn't received the update yet.
The system scaled.
Consistency became more complicated.
Problem
↓
Solution
↓
New Trade-off
Stage 8 --- 750,000 Users: Stop Sending Everything Through the Backend
Images, JavaScript bundles, CSS, downloads, and other static assets
don't necessarily need to travel through application servers on every
request.
A CDN can move cacheable content closer to users.
Users
│
┌───────┴────────┐
▼ ▼
CDN Load Balancer
│ │
Static Content API Servers
This reduces:
- Origin traffic
- Backend bandwidth
- Static-content latency
- Unnecessary application-server work
Again, the CDN has a reason to exist.
Stage 9 --- 1 Million Users: Stop Doing Everything Synchronously
Consider what happens when someone creates a booking.
Our API might perform:
Create Booking
↓
Send Email
↓
Send Push Notification
↓
Update Analytics
↓
Award Loyalty Points
↓
Notify Professional
↓
Return Response
The customer is waiting for work that doesn't necessarily need to finish
before the booking is acknowledged.
The core path may only require:
Validate
↓
Reserve
↓
Persist
↓
Confirm
Other work can happen asynchronously:
Booking Service
│
▼
BookingCreated
│
▼
Event Broker
│
┌───────────┼───────────┐
▼ ▼ ▼
Notify Analytics Loyalty
This improves responsiveness and decouples downstream work.
But now we need to think about:
- Duplicate events
- Consumer failures
- Ordering
- Retries
- Dead-letter queues
- Database/event consistency
For example:
Database COMMIT ✓
Event publish ✗
That's not just a messaging problem.
It's a consistency problem.
We'll explore that later in the series.
What Did We Actually Build?
We started here:
User → Application → Database
And ended somewhere closer to this:
Users
│
▼
CDN
│
▼
Load Balancer
│
┌──────────┼──────────┐
▼ ▼ ▼
API 1 API 2 API 3
│ │ │
└──────────┼──────────┘
│
┌───────┴────────┐
▼ ▼
Cache Database
│
┌────────┴────────┐
▼ ▼
Replica 1 Replica 2
│
▼
Event Broker
│
┌─────────┼─────────┐
▼ ▼ ▼
Worker Notify Analytics
That's a much more sophisticated architecture.
But here's the important part:
We didn't start by designing this architecture. We arrived at it.
Every component earned its place.
One Million Users Didn't Create One Scaling Problem
It created a sequence of different problems.
Growth Stage What Starts Hurting Architectural Response
1--100 Nothing significant Keep it simple
100--10K Inefficient Optimize and scale
code/queries, resource vertically
pressure
10K--50K Single-server capacity Horizontal scaling +
load balancing
50K--100K Instance-local state Stateless services /
shared state
100K--250K Repeated expensive Caching
reads
250K--500K Database pressure Query optimization +
read scaling
500K--750K Origin/static-content CDN / edge delivery
load
750K--1M Too much synchronous Async processing /
work event-driven workflows
These user counts are illustrative, not universal thresholds.
A read-heavy application may hit database limits much earlier.
A compute-heavy application may hit CPU limits first.
A media platform may need a CDN almost immediately.
A financial system may prioritize consistency and transaction boundaries
long before raw traffic becomes interesting.
Scale is workload-specific.
Don't Scale Users. Scale Bottlenecks.
You don't actually scale because you reached:
100,000 users
You scale because something measurable changed:
CPU ↑
Memory ↑
Connections ↑
Queue depth ↑
Database latency ↑
Error rate ↑
p95 / p99 ↑
User count is context.
Resource pressure and system behavior tell you what needs to change.
Two applications with one million users can require completely different
architectures.
The Scaling Loop
The mental model from Part 0 still applies:
Traffic grows
↓
Observe
↓
Find bottleneck
↓
Understand cause
↓
Choose solution
↓
Measure again
↓
Repeat
Not:
Traffic grows
↓
Add every technology we know
That difference is where architecture starts becoming engineering.
The Real Goal
The goal isn't to build a one-million-user architecture on day one.
The goal is to build a system that:
- Works for today's requirements
- Is observable enough to tell you what is breaking
- Has clear boundaries where change is likely
- Can evolve when the next constraint appears
A three-box architecture that correctly serves your current workload is
better than a fifteen-service architecture whose complexity you don't
need.
Start simple.
Measure.
Find the bottleneck.
Solve that bottleneck.
Then repeat.
Don't design for one million users on day one. Design a system that
gives you a clear path to the next stage.
Up Next
We've spent this article scaling application infrastructure.
But eventually, almost every system-design discussion reaches another
question:
What database should we actually use?
SQL? NoSQL? Document? Key-value? Graph?
More importantly:
How do we choose without starting from technology hype?
Next → Part 2
How to Choose Between SQL, NoSQL, and Everything in Between
This is **Part 1* of System Design from Developer to Architect ---
a practical series about scalability, databases, APIs, distributed
systems, reliability, and the engineering decisions behind production
architecture.*
Previous: Part 0 --- How to Think About System Design Without Just
Drawing
Boxes
Top comments (0)