DEV Community: HappyBug

System Design the Agentic Way — Phase 2: Stateless Horizontal Scaling

HappyBug — Tue, 14 Jul 2026 11:09:21 +0000

Phase 2: Stateless Horizontal Scaling

Phase 1 ended at a wall: a single machine has hard ceilings, and no amount of better code moves them. The obvious answer is "add more servers." Phase 2 is about everything that sentence quietly assumes — and the new failure modes that appear the moment a second machine exists.

The Question
The Building Blocks: What "Add a Server" Actually Requires
- State: The Thing That Breaks First
- The Shared-Nothing Principle
- Sticky Sessions: The Tempting Trap
- Load Balancing Algorithms
- Layer 4 vs Layer 7
- Latency vs Throughput
Concurrent Requests Don't Run Sequentially
What the Experiments Showed
The Fix Reveals the Lesson
What This Phase Actually Teaches
Where I'm Still Fuzzy
Try It Yourself
Next

The Question

You have one server and you want three. What has to be true about your application before adding the other two actually helps — and where does the bottleneck go once you do?

Phase 2 is an argument that "add more servers" is not an infrastructure task. It's an application-architecture constraint that decides whether the extra machines do anything at all.

The Building Blocks: What "Add a Server" Actually Requires

A war story sets the stakes. A food delivery startup ran its entire API on one server. A routine Friday 6pm deploy meant an 8-second restart. During those 8 seconds: 47 lost payment confirmations, 312 connection resets, and — because the process had to reload a 2.3 GB data.json — a 45-second full recovery. The bill was $23,000 in refunds.

The root cause isn't the deploy. It's that a single machine can't be restarted without downtime and can't survive its own hardware dying. The fix is to run several identical copies. But copies only work if the servers hold no state — and that word hides most of the difficulty.

State: The Thing That Breaks First

Take the Phase 1 server and imagine running three copies of it. Three distinct kinds of state each break in their own way:

Type of state	Example	Why 3 copies break it
Persistent data	`data.json` on disk	Each server has its own copy → writes diverge, reads disagree
In-memory counters	`activeConnections`	Each server sees only its own slice → no global view
Coordination state	write queue / locks	Only valid inside one process → no cross-server ordering

The pattern: anything a server remembers between requests becomes wrong the moment a second server exists, because the next request might land somewhere else.

The Shared-Nothing Principle

The fix is to externalize all state to a shared service — a database, a cache like Redis, a metrics store like Prometheus — so that no server owns anything. The server becomes a pure function:

request in → (read shared state) → (write shared state) → response out
server remembers NOTHING between requests

This is the shared-nothing architecture. Any request can go to any server, because every server is interchangeable. That interchangeability is the entire point — it's what makes servers disposable, and disposable servers are what let you deploy, restart, and lose hardware without an outage.

Sticky Sessions: The Tempting Trap

If a server does hold per-user state (a login session in memory), you can paper over it by pinning each user to the same server every time — a sticky session.

Stateful  + sticky sessions → user X always routed to Server 2
Stateless + any routing     → user X can be served by any server

The trap: sticky sessions reintroduce the single-machine failure. When Server 2 dies, every user pinned to it loses their session — logged out, cart emptied — even though two other servers are healthy. Sticky sessions are a temporary crutch. The goal is always full statelessness, where a dying server costs nothing.

A related temptation is to jump straight to microservices "to reduce blast radius." Scale the monolith horizontally first: three identical copies, deployed one at a time (a rolling deploy), gives zero-downtime with zero architectural change.

Monolith, 1 server      → single point of failure
Monolith, 3 servers     → scales, survives restarts, still SIMPLE
Microservices, 20 svcs  → scales, but adds network failures + distributed transactions

Decompose only when there's a specific reason to (that's Phase 9), not reflexively.

Load Balancing Algorithms

With three interchangeable servers, something has to decide who gets each request. That's the load balancer, and the decision rule matters:

Algorithm	How it works	Best for	Weakness
Round Robin	A, B, C, A, B, C…	Equal servers, equal-cost requests	Ignores how heavy each request is
Least Connections	Route to server with fewest active connections	Mixed request durations	Blind to raw server capacity
Weighted Round Robin	More traffic to beefier servers	Heterogeneous hardware	Static, doesn't adapt to load
Random	Pick any server	Surprisingly good at scale	Can cluster at low request counts

For a server where /heavy takes 5 s but /data/:key takes 5 ms, Least Connections is the right pick. Round Robin would keep handing new requests to a server already stuck on a slow one:

Round Robin:  Server A busy on /heavy (5s) → still gets 2 more → queue builds
Least Conn:   Server A shows 3 active     → skip it → route to Server B (0 active) ✓

Layer 4 vs Layer 7

"Reverse proxy" and "load balancer" are not separate things — any proxy that hides backends can trivially point at several of them. The meaningful distinction is which network layer it operates on:

Layer 4 (transport):  routes packets by IP + port. Fast, protocol-blind.
                      Example: AWS Network Load Balancer.
Layer 7 (application): understands HTTP — can route by path, header, cookie.
                      Example: Nginx, HAProxy, Envoy.

Layer 7 costs a little more per request but can make smart, content-aware routing decisions. This prototype uses Nginx (Layer 7).

Latency vs Throughput

Externalizing state has a cost, and naming it correctly matters. Two independent measures:

Latency    = how long ONE request takes         (a single car driving A → B)
Throughput = how many requests complete per unit (cars arriving per hour)

Adding servers (or highway lanes) does not make a single request faster. It lets the system handle more requests at once. Concretely, moving from a local file to an external store:

	Phase 1 (local file)	Phase 2 (external store)
Latency	~0.2 ms	~5 ms (25× slower)
Throughput	~5K req/s	~30K req/s (6× more)
Failure tolerance	zero	survives a server dying

The trade is deliberate: accept higher per-request latency in exchange for far higher throughput and the elimination of the single point of failure. As long as latency stays under the ~100 ms human-perception threshold, users don't notice — but they very much notice an outage.

Concurrent Requests Don't Run Sequentially

A common misread: "if each request now waits 5 ms on the database, 10,000 requests take 50 seconds." That's wrong, and understanding why is the crux of the phase.

An await yields the event loop. While one request waits on the database, the server starts the next:

t=0.00 ms  Req 1 arrives → await db.read() → YIELDS
t=0.01 ms  Req 2 arrives → await db.read() → YIELDS
t=0.02 ms  Req 3 arrives → await db.read() → YIELDS
t=2.00 ms  Req 1's response comes back → resume → reply
t=2.01 ms  Req 2's response comes back → resume → reply

Each request individually experiences 5 ms. They overlap in time. The system does not fall over from per-request latency — it falls over when the database saturates:

3 servers × 10K req/s × 2 DB ops each = 60,000 DB ops/second

And that is the real lesson of horizontal scaling:

BEFORE:  [Server] ← ceiling was here (Phase 1: ~15K connections)

AFTER:   [Server 1] ─┐
         [Server 2] ─┼─→ [Database] ← ceiling moved HERE
         [Server 3] ─┘

Scaling doesn't eliminate bottlenecks. It moves them downstream.

What the Experiments Showed

The prototype makes this concrete: the Phase 1 server rewritten as an "amnesia server" (all fs calls replaced with Redis get/set), packaged into one Docker image, run as three containers behind an Nginx load balancer, all sharing one Redis.

Client → Nginx (:80) → server1:3000 ─┐
                       server2:3000 ─┼─→ Redis (:6379)  ← shared state
                       server3:3000 ─┘

Every response includes served_by: os.hostname() to prove distribution. Three verifications passed first:

Load distribution — 10 parallel /stats requests returned 3 different hostnames (a 4-3-3 split).
Shared state — a PUT through one server was immediately readable through the other two. Redis is the single source of truth.
Server death — docker stop server1, then read: the remaining two servers served all data with zero loss. The server is genuinely disposable.

Then came breaking it on purpose.

Experiment 1 — Kill the shared state: a hang is worse than a crash

docker stop on Redis, then a request. The expectation was an error response. What actually happened:

curl → server → await redis.get() → (never resolves) → request HANGS

No crash, no error — the Redis client silently retries the connection forever, so await redis.get() never settles. This is the worst failure mode: a crash returns an error instantly, but a hang ties up the connection, and enough hangs exhaust Nginx's connection pool and freeze the entire system. Always set timeouts on external service calls. A fast failure is recoverable; a silent hang cascades.

Experiment 2 — Restart Redis: did the data survive?

docker start on Redis, then read the old keys. The data was still there. Redis writes an RDB snapshot to disk on a graceful shutdown (SIGTERM).

The caveat matters: a docker kill (SIGKILL) or a real crash skips the snapshot, so everything since the last auto-save is lost. Graceful shutdown preserved durability here; an ungraceful one wouldn't have. Real durability guarantees are a Phase 3 topic.

Experiment 3 — Kill the load balancer: the new single point of failure

docker stop on Nginx, then curl localhost: connection refused. The servers and Redis were all healthy — reachable directly via docker exec — but unreachable from outside, because Nginx is the only public entry point.

Client → [Nginx]  → Server 1 → [Redis]
          SPOF       Server 2    SPOF
                     Server 3
                    (redundant)

Only the servers gained redundancy. Scaling moved the single point of failure (SPOF) from the server to two new places: the load balancer in front and the shared state behind.

The Fix Reveals the Lesson

Each break has a distinct remedy, and each remedy points at a later phase:

State divergence across servers → externalize every kind of state to a shared service; keep the server a pure function.
Hang on dependency failure → wrap every external call in a timeout (and eventually a circuit breaker — Phase 11).
Load balancer / shared-state SPOF → run multiple load balancers behind a cloud LB, and replicate the shared state (Phase 6).

The surface symptoms differ, but the shape is identical to Phase 1: the component that fails determines both the symptom and the fix. What changed is that "the component" is now something downstream of the server you added — which is exactly what horizontal scaling does to bottlenecks.

What This Phase Actually Teaches

1. Horizontal scaling is an application property, not an infrastructure one. Adding servers only helps if the servers are stateless. The hard work is externalizing state (data, counters, coordination); provisioning the machines is the easy part.

2. Shared-nothing makes servers disposable, and disposable is the whole goal. When any request can go to any server, you can deploy one at a time, tolerate hardware death, and restart freely — the three things a single machine can never do.

3. Sticky sessions quietly reintroduce single-machine failure. Pinning users to servers is a crutch; a dying server takes its pinned users down with it. Full statelessness is the target.

4. The load-balancing algorithm must match request-cost variance. With uniform requests, Round Robin is fine. With a mix of 5 ms and 5 s requests, Least Connections avoids piling work onto an already-busy server.

5. Scaling moves bottlenecks downstream — it never deletes them. Three servers turn one ceiling (connections) into a new one (database throughput at ~60K ops/s) plus two new SPOFs (the LB and the shared state). Know where the ceiling went.

6. A hang is worse than a crash. An unbounded wait on a dead dependency ties up resources and cascades. Every external call needs a timeout.

Decision heuristic:

Need zero-downtime deploys / HA, low complexity   → monolith × N behind a load balancer
Requests vary wildly in cost                      → Least Connections
Uniform requests, equal servers                   → Round Robin
Need path/header/cookie-aware routing             → Layer 7 (Nginx/Envoy)
Need raw packet throughput, protocol-blind        → Layer 4 (cloud NLB)
Independent scaling / team boundaries force it     → microservices (Phase 9), not before

Where I'm Still Fuzzy

Timeout values on external calls. I know an unbounded wait is the failure — but what's the right timeout? Too low and you fail healthy-but-slow requests; too high and you hang. It presumably depends on the dependency's p99, but I don't have a principled way to set it yet.
Load balancer redundancy in practice. "Run multiple Nginx behind a cloud LB" is the stock answer, but that just pushes the SPOF up to the cloud LB. Where does the regress actually stop — DNS? Anycast? I haven't built the layer above Nginx.
When Least Connections misleads. It's blind to server capacity, so a weak server with few connections still looks attractive. In a heterogeneous fleet, when does that go wrong, and is weighted-least-connections the real answer?

Try It Yourself

The prototype is at github.com/aishwarya-chamanoor/system-design-agenticway in phase-02/.

git clone https://github.com/aishwarya-chamanoor/system-design-agenticway
cd system-design-agenticway/phase-02
docker compose up

Four experiments worth running:

Hit http://localhost/stats ten times in parallel — watch served_by cycle across three hostnames.
PUT a value through one request, then GET it repeatedly — every server returns the same value because Redis owns it.
docker stop <server-container> and keep reading — the surviving servers serve everything; the server is disposable.
docker stop <redis-container> and send one request — watch it hang instead of erroring. That hang is the lesson.

Phase 3 goes inside the shared store this phase leaned on — asking what "the data survived" really guarantees, and what it costs to make that guarantee real.

Part of a 12-phase series building distributed systems intuition from first principles. Each phase has a running prototype, failure scenarios, and a gate check before moving on.

System Design the Agentic Way — Phase 1: The Single Machine Ceiling

HappyBug — Mon, 13 Jul 2026 13:34:05 +0000

Phase 1: The Single Machine Ceiling

Before reasoning about distributed systems, you need a concrete model of what a single machine actually runs out of. Phase 1 builds that model from the ground up — not from diagrams, but from pushing a server until it stops accepting connections, and understanding exactly what gave out and why.

The Question
The Building Blocks: What Actually Handles a Connection
- File Descriptors
- TCP Connections: The Handshake
- Keep-Alive: Efficiency With a Hidden Cost
- TIME_WAIT: The Port Cooldown
The Resource Landscape
Concurrency Models: How the Server Handles Multiple Users
- Thread-per-Connection
- Event Loop
- Thread Pool
- The Comparison That Matters
- The "Pure Event Loop" Lie
What the Experiments Showed
The Fix Reveals the Lesson
What This Phase Actually Teaches
Where I'm Still Fuzzy
Try It Yourself
Next

The Question

A server has CPU, memory, disk, and network capacity. When it fails under load, which one ran out — and does the answer change what you do about it?

Phase 1 is an argument that it matters enormously, and that most engineers are watching the wrong gauges.

The Building Blocks: What Actually Handles a Connection

Before getting to failure modes, it helps to have a concrete model of what happens when a user connects to a server.

File Descriptors

The OS tracks every open resource — files, network connections, pipes — using a small integer called a file descriptor (FD). The first three are always reserved:

FD 0 → stdin  (keyboard input)
FD 1 → stdout (console output)
FD 2 → stderr (error output)
FD 3 → your log file
FD 4 → user A's connection
FD 5 → user B's connection
...
FD 1024 → OS says "EMFILE — no more" ❌

Every TCP connection your server accepts consumes one FD. The default per-process limit on Linux is 1,024. When that fills, the OS returns EMFILE (too many open files) and refuses new connections — regardless of how much CPU or memory you have left.

$ ulimit -n        # check current limit → probably 1024
$ ulimit -n 65535  # raise it (current session only)
# Permanent: edit /etc/security/limits.conf

TCP Connections: The Handshake

Every HTTP connection starts with a TCP three-way handshake before any data flows:

Client → Server: SYN      ("I'd like to connect")
Server → Client: SYN-ACK  ("Got it, I'm ready")
Client → Server: ACK       ("Confirmed — send your request")
Client → Server: [data]    (the actual HTTP request)

Each step takes a network round-trip. For a client in the same data center that's ~0.1ms; across continents it's 100–300ms. This overhead motivated HTTP Keep-Alive.

Keep-Alive: Efficiency With a Hidden Cost

HTTP Keep-Alive reuses a single TCP connection for multiple requests instead of handshaking for each one:

Without: handshake → request → close,  handshake → request → close  (2 handshakes)
With:    handshake → request → request → request → close             (1 handshake)

Faster for the client. But idle keep-alive connections still hold an FD open on the server doing nothing. 5,000 users sitting on an idle browser tab = 5,000 FDs consumed. This is a trap that ended the startup in the war story below.

TIME_WAIT: The Port Cooldown

When a TCP connection closes, the port on the client side enters TIME_WAIT for 60–120 seconds. During this window, that port can't be reused for new outbound connections.

1,000 outbound connections/sec × 60 sec TIME_WAIT = 60,000 ports locked
Total available ephemeral ports: ~65,000
→ Port exhaustion, even with healthy CPU and memory

This matters most on servers that make many outbound connections (proxies, services that call many APIs). You'll see connect: Cannot assign requested address in logs.

The Resource Landscape

A server has five independent ceilings, each with its own failure signature:

Resource	What fills it	Failure signature	Default limit
File descriptors	TCP connections, open files	`EMFILE` — new connections refused	1,024/process
Memory	Heap, thread stacks, buffers	OOM Killer silently kills process	Depends on RAM
CPU	Computation, context switching	All requests slow down together	100% across cores
Disk IOPS	Read/write operations	Write latency spikes	SSD: ~100K–500K/s
Network ports	Outbound connections (TIME_WAIT)	New outbound connections fail	~65K ephemeral ports

The critical distinction: CPU and memory saturation cause gradual degradation — everything slows down proportionally. FD and port exhaustion cause total failure — new connections are refused while existing ones and dashboard metrics look completely healthy. The second kind is the one that produces a 47-minute outage where nobody can figure out why the server is "down."

Concurrency Models: How the Server Handles Multiple Users

How a server handles multiple simultaneous connections determines which ceiling it hits first. There are three main approaches.

Thread-per-Connection

A new OS thread is created for each incoming connection. That thread handles everything for that connection — reading the request, processing it, writing the response — then terminates.

User A connects → Thread 1 created (1MB RAM)
User B connects → Thread 2 created (1MB RAM)
User C connects → Thread 3 created (1MB RAM)
...
User 32,000 → Thread 32,000 → OOM

Simple to code: each thread runs straight-line blocking code. But each thread costs ~1MB of stack space, and context switching between thousands of threads burns CPU. At 10,000 threads, 10–30% of CPU goes just to saving and restoring thread state. At 100,000 — the system is unusable.

First ceiling: Memory (32GB ÷ 1MB/thread ≈ 32,000 max threads), then CPU from context switching.

Event Loop

One thread handles all connections by processing callbacks as they become ready. Instead of blocking while waiting for I/O, it registers a callback and moves on.

Hotel receptionist analogy:
  Loop forever:
    1. Any new guests?        → start check-in
    2. Any responses ready?   → send them
    3. Any room service done? → notify the room (non-blocking)
    4. Back to step 1

ONE receptionist. MANY guests. Never stands idle waiting.

Each connection costs only a few KB (just a file descriptor and some socket state), not 1MB. The same 32GB server can hold 100,000+ connections.

First ceiling: File descriptors (OS limit), or CPU if any one request does synchronous computation.

Weakness: Head-of-line blocking. One slow synchronous task holds the single thread, freezing every other connection behind it.

Drive-through analogy:
  Car A: "Custom birthday cake" (10 min)
  Car B: "Just a coffee"  ← waits 10 minutes behind Car A
  Car C: "A muffin"       ← waits 10:30

Thread Pool

A fixed number of pre-created threads pull work from a shared queue. Balances the two extremes: bounded memory cost (N threads, not one per connection), and true parallelism across CPU cores.

Hire 20 waiters. All new tables wait in the lobby.
A free waiter → grabs next table → serves → returns to lobby.
If all 20 are busy → table waits in queue.
If queue is full → reject with 503.

The pool size formula:

pool_size = num_cores × (1 + wait_time / compute_time)
Example: 8 cores, 90ms DB wait, 10ms compute → 8 × 10 = 80 threads

Failure mode: If a slow downstream dependency holds all threads (e.g., 200 threads × 30-second timeout = all blocked), the queue fills and the service rejects everything — even though your code is fine.

The Comparison That Matters

	Thread-per-Connection	Event Loop	Thread Pool
Memory @ 10K connections	~10 GB	~50 MB	~500 MB
Max practical connections	1K–10K	10K–100K+	10K–50K
Failure mode	OOM / context-switch death spiral	Head-of-line blocking	Queue overflow
Race conditions	Every CPU instruction boundary	Only at `await` boundaries	Every CPU instruction boundary
Who uses it	Apache httpd	Node.js, nginx, Redis	Java Tomcat, Go/Netty

The "Pure Event Loop" Lie

Node.js is described as single-threaded. It isn't, exactly. The event loop handles network I/O directly (non-blocking), but file I/O and DNS lookups are blocking operations that can't be done asynchronously by the OS. Node offloads these to a small internal thread pool via libuv:

Event loop handles:          libuv thread pool (default: 4 threads) handles:
  TCP connections              fs.readFile / writeFile
  HTTP parsing                 DNS lookups (dns.lookup)
  DB queries over network      crypto (pbkdf2, scrypt)
  setTimeout / setInterval     zlib compression

The 4-thread default is dangerously low in production. If 4 file reads are in progress, a DNS lookup queues behind them. HTTP calls to external APIs start timing out for no obvious reason. The fix: UV_THREADPOOL_SIZE=64 set before the process starts.

Every "event loop" runtime is actually a hybrid. Knowing which operations go to the thread pool and which stay on the event loop is essential for debugging latency spikes.

What the Experiments Showed

The prototype: a Node.js HTTP key-value store, raw http module, zero npm packages. Four routes: /data/:key (read/write), /stats (live metrics — memory, FDs, event loop lag), and /heavy (a deliberate CPU blocker). Simple enough to break in predictable ways.

Experiment 1 — Connection flood: the ceiling was the OS kernel

Opening TCP connections without closing them, stepping up in batches. The server stayed alive until:

15,965 connections → ENOBUFS (no buffer space available)
CPU: ~0%    Node heap: ~480 MB (healthy)

The event loop was not blocked. Node's memory was fine. What ran out was OS kernel network buffer space — roughly 250 MB of non-paged pool memory consumed by socket receive/send buffers (~16 KB each). The ceiling was in the kernel, not in the application.

Ceiling hierarchy hit in order:
  1. OS kernel network buffers  ← actual ceiling on Windows
  2. FD limit (~16K on Windows) ← would have been next
  3. Node.js heap               ← never reached
  4. CPU                        ← never reached

This is the war story that opens the phase: a social media API during a viral spike. CPU at 5%, memory at 40%, every dashboard looked healthy — but the server was refusing every new connection. 47-minute outage. The fix:

ulimit -n 65535   # raises the per-process FD limit from the default 1,024

The point isn't to memorize ulimit. It's that the server had massive headroom on four of its five ceilings, and failed on the fifth one nobody was measuring.

Experiment 2 — One CPU-bound request freezes all 15,965 users

A single request to /heavy:

// This holds the event loop thread — nothing else can run
const start = Date.now();
while (Date.now() - start < 5000) {}

During those 5 seconds:

/stats response time:  5,000 ms  (normally 1 ms)
Event loop lag:        12,000 ms
CPU:                   98%
All other requests:    frozen

The /stats route does trivial work — read process memory, count handles, respond. Under normal conditions it takes 1ms. The while loop on /heavy blocked it for 5 full seconds because the event loop is a single thread with cooperative scheduling. There is no OS scheduler interrupting a running synchronous task to let another request proceed. The CPU-bound route held the thread until it finished.

This surfaces in production from: JSON.parse() on a 50 MB payload, an unanchored regex on user input (ReDoS), or a forgotten fs.readFileSync() in a request handler. All of them stall every connected user simultaneously.

Experiment 3 — Async doesn't make reads safe

100 concurrent PUT requests, each reading the current counter, incrementing it, and writing it back. Expected final value: 100. Actual: 60-something.

The interleaving that causes this:

Request A: await readStore()   → gets { counter: 1 }  ← yields here
Request B: await readStore()   → gets { counter: 1 }  ← stale; A hasn't written
Request A: await writeStore({ counter: 2 })
Request B: await writeStore({ counter: 2 })            ← overwrites; should be 3

The assumption going in: single-threaded event loop means no interleaving. That's true between await calls — code between two awaits is atomic. But the full async function is not. Every await is a yield point where the event loop can run another callback. In a read-modify-write pattern, that's enough.

The same lost-update problem exists in multi-threaded code, just at every CPU instruction instead of only at await boundaries. The event loop makes races less frequent, not impossible.

The Fix Reveals the Lesson

Each experiment has a distinct fix:

FD/buffer exhaustion → raise ulimit -n / set nofile in systemd unit or container config
Event loop blocking → offload synchronous CPU work to worker_threads; never call blocking APIs in a hot path
Async race conditions → serialize writes through a queue, or use atomic DB operations (transactions, compare-and-swap)

All three experiments produced similar surface symptoms — slow or unresponsive server. The root causes are completely different. Applying the wrong intervention (or adding a second server) wouldn't have helped. The resource that fails determines both the symptom and the remedy.

What This Phase Actually Teaches

Not all resource ceilings behave like slowness. FD and port exhaustion produce hard refusals while CPU and memory metrics look healthy. This class of failure is disproportionately misdiagnosed because teams are watching the wrong gauges.

The concurrency model determines which ceiling you hit first. Thread-per-connection hits memory first (~1MB/thread → ~32K threads on 32GB RAM). Event-loop hits file descriptors or CPU-bound blocking first. The model isn't just a performance choice — it changes the failure mode.

What This Phase Actually Teaches

1. A server has five independent resource ceilings, and the smallest one is the actual limit. CPU saturation causes gradual degradation. FD exhaustion and port exhaustion cause total failure while all other metrics look fine. Know which ceiling you're approaching before an incident.

2. The thread-per-connection model kills itself on memory and context switching. At 10,000 threads: ~10 GB of stack space, and 10–30% of CPU wasted on saving/restoring thread state. This is why Node.js, nginx, and Redis all chose the event loop.

3. The event loop trades memory efficiency for CPU vulnerability. A single synchronous task holds the thread. Head-of-line blocking isn't a bug to fix — it's a fundamental property of cooperative concurrency. The architecture choice is: which failure mode can I tolerate? OOM death spiral, or one hot request freezing everyone?

4. "Single-threaded" is a simplification. Node.js has 4 libuv worker threads by default handling file I/O, DNS, and crypto. DNS lookups queue behind file reads. In production, set UV_THREADPOOL_SIZE=64.

5. Async code has race conditions at await boundaries. Code between two awaits is atomic. A full async function is not. Every read-modify-write pattern across awaits needs explicit serialization.

When to use which model:

< 5K connections, simple code needed        → Thread-per-connection
> 10K connections, mostly I/O-bound         → Event loop
Mixed CPU + I/O, need parallelism           → Thread pool
Production at scale                         → Hybrid (event loop + worker pool)

Where I'm Still Fuzzy

ENOBUFS vs EMFILE — I hit ENOBUFS at ~16K connections on Windows, which maps to OS kernel buffer exhaustion, not the FD limit. On Linux with default ulimit -n 1024, EMFILE would appear much earlier. Are these always separable in production?
TIME_WAIT at high connection churn — I understand port exhaustion in theory (1K conns/sec × 60s = 60K locked ports), but I haven't built something that actually hits it. That would require a high-outbound-connection scenario — probably relevant in Phase 9 (service decomposition) when services call each other.
libuv thread pool saturation in practice — the 4-thread default is clearly too low, but what does the performance curve look like as you raise it? Is there a point where more threads hurt?

Four experiments worth running:

GET /stats — see live FD count, heap usage, and event loop lag
Hit /heavy in one tab, then hit /stats from another — watch the lag number
Run 100 concurrent writes to a counter and check if the final value matches the expected count
Open thousands of connections without closing them and watch when the OS complains first

The stats endpoint measures event loop lag by scheduling a setTimeout(0) callback and measuring how long before it actually runs. Under normal load: < 5ms. During a CPU-bound request: in the thousands.

Phase 2 moves the problem to multiple machines — which immediately surfaces a new class of failures that single-machine thinking doesn't predict.

Part of a 12-phase series building distributed systems intuition from first principles. Each phase has a running prototype, failure scenarios, and a gate check before moving on.

System Design the Agentic Way — How I'm Using AI Agents to Actually Learn Distributed Systems

HappyBug — Mon, 13 Jul 2026 06:30:28 +0000

System Design the Agentic Way

I wanted to learn system design. Not "memorize CAP theorem for interviews" learn — actually understand why things break at scale and what to do about it.

The problem: most resources give you theory without pain. You read about load balancers without ever watching one fail. You learn about replication without seeing stale data appear in front of you.

So I built something different.

The Idea

What if I had a pair-programmer that:

Starts every topic with a real production failure (so I feel why it matters)
Makes me predict what will happen before I run anything
Forces me to build minimal prototypes, not just read
Then makes me break them deliberately
And finally tests my judgment with ambiguous 3am scenarios

That's what I built — using GitHub Copilot's custom agents and skills as a structured teaching system.

The Curriculum

12 phases. Each one builds on the previous. No skipping.

Phase	What Goes Wrong Without It
1. Single Machine Ceiling	Your server hits 15K connections and starts refusing
2. Stateless Horizontal Scaling	You deploy and lose 47 payments during the restart
3. Data Storage Trade-offs	Redis crashes and your last 30 seconds of writes vanish
4. Caching & Invalidation	Users see stale prices and buy at the wrong amount
5. Async Processing & Queues	Payment service is down for 2 seconds, 500 orders lost
6. Replication & Consistency	Read replicas serve yesterday's data as if it's current
7. Partitioning & Sharding	One customer has 90% of the data, hot partition melts
8. Coordination & Consensus	Two nodes both think they're the leader, split-brain
9. Service Decomposition	Distributed transaction fails halfway, money vanishes
10. Observability	Something is wrong. You have no idea what.
11. Resilience Patterns	One slow dependency freezes your entire service
12. Capacity Planning	Black Friday hits 10x predictions, everything collapses

The Method

Every phase follows the same loop:

Motivate → Model → Decide → Build → Break → Gate

🔥 War Story    → Feel the pain
🧠 Concepts     → Understand the mechanism
⚖️ Trade-off   → Pick an approach (and own the downsides)
🔨 Build       → Minimal prototype with real code
💥 Break It    → Deliberately kill components
🎯 Gate Check  → Scenario-based judgment test

The "Break It" step is where most of the learning happens. My prediction is almost always wrong the first time. That gap — between what I expected and what actually happened — is the lesson.

The Tooling

The whole system runs inside VS Code using GitHub Copilot's custom agent features:

5 agents — instructor, gate-keeper, adversary, reviewer, orchestrator
8 skills — war stories, concept deep-dives, prototype specs, failure labs, etc.
Knowledge files — real production patterns, failure catalogs, anti-patterns
Learning journal — auto-captured insights, predictions, and mistakes

The agents don't give me answers. They ask me to predict, let me fail, then explain why.

Where I Am

I've completed Phase 1 (single machine limits) and Phase 2 (horizontal scaling with Docker + Nginx + Redis). In the next posts, I'll walk through each phase — what I built, what I broke, and what surprised me.

Try It Yourself

The full system is open source:

👉 system-design-agenticway

What I'd Love From You

⭐ Star the repo if this approach resonates
🐛 Open issues if something's broken or confusing
💡 Suggest improvements to the curriculum
📝 Tell me: how do YOU learn system design? What works? What doesn't?

Next up: Phase 1 — The Single Machine Ceiling. I'll show you what happens when 15,965 TCP connections hit one Node.js process, and what it taught me about operating systems.

Follow for the full series. Each post covers one phase of the journey.

Clone it, open in VS Code with Copilot, and start Phase 1. All you need is Node.js and Docker.

What I'd Love From You

⭐ Star the repo if this approach resonates
🐛 Open issues if something's broken or confusing
💡 Suggest improvements to the curriculum
📝 Tell me: how do YOU learn system design? What works? What doesn't?

Next up: Phase 1 — The Single Machine Ceiling. I'll show you what happens when 15,965 TCP connections hit one Node.js process, and what it taught me about operating systems.

Follow for the full series. Each post covers one phase of the journey.

How are people collaborating here for dev challenges? Need advice :)

HappyBug — Thu, 04 Jun 2026 09:18:12 +0000

DEV Community: HappyBug

System Design the Agentic Way — Phase 2: Stateless Horizontal Scaling

Phase 2: Stateless Horizontal Scaling

Table of Contents

The Question

The Building Blocks: What "Add a Server" Actually Requires

State: The Thing That Breaks First

The Shared-Nothing Principle

Sticky Sessions: The Tempting Trap

Load Balancing Algorithms

Layer 4 vs Layer 7

Latency vs Throughput

Concurrent Requests Don't Run Sequentially

What the Experiments Showed

Experiment 1 — Kill the shared state: a hang is worse than a crash

Experiment 2 — Restart Redis: did the data survive?

Experiment 3 — Kill the load balancer: the new single point of failure

The Fix Reveals the Lesson

What This Phase Actually Teaches

Where I'm Still Fuzzy

Try It Yourself

Next

System Design the Agentic Way — Phase 1: The Single Machine Ceiling

Phase 1: The Single Machine Ceiling

Table of Contents

The Question

The Building Blocks: What Actually Handles a Connection

File Descriptors

TCP Connections: The Handshake

Keep-Alive: Efficiency With a Hidden Cost

TIME_WAIT: The Port Cooldown

The Resource Landscape

Concurrency Models: How the Server Handles Multiple Users

Thread-per-Connection

Event Loop

Thread Pool

The Comparison That Matters

The "Pure Event Loop" Lie

What the Experiments Showed

Experiment 1 — Connection flood: the ceiling was the OS kernel

Experiment 2 — One CPU-bound request freezes all 15,965 users

Experiment 3 — Async doesn't make reads safe

The Fix Reveals the Lesson

What This Phase Actually Teaches

What This Phase Actually Teaches

Where I'm Still Fuzzy

Next

System Design the Agentic Way — How I'm Using AI Agents to Actually Learn Distributed Systems

System Design the Agentic Way

The Idea

The Curriculum

The Method

The Tooling

Where I Am

Try It Yourself

What I'd Love From You

What I'd Love From You

How are people collaborating here for dev challenges? Need advice :)