System Design the Agentic Way — Phase 1: The Single Machine Ceiling

Phase 1: The Single Machine Ceiling

Before reasoning about distributed systems, you need a concrete model of what a single machine actually runs out of. Phase 1 builds that model from the ground up — not from diagrams, but from pushing a server until it stops accepting connections, and understanding exactly what gave out and why.

---

Table of Contents

• The Question
• The Building Blocks: What Actually Handles a Connection
  • File Descriptors
  • TCP Connections: The Handshake
  • Keep-Alive: Efficiency With a Hidden Cost
  • TIME_WAIT: The Port Cooldown
• The Resource Landscape
• Concurrency Models: How the Server Handles Multiple Users
  • Thread-per-Connection
  • Event Loop
  • Thread Pool
  • The Comparison That Matters
  • The "Pure Event Loop" Lie
• What the Experiments Showed
• The Fix Reveals the Lesson
• What This Phase Actually Teaches
• Where I'm Still Fuzzy
• Try It Yourself
• Next

---

The Question

A server has CPU, memory, disk, and network capacity. When it fails under load, which one ran out — and does the answer change what you do about it?

Phase 1 is an argument that it matters enormously, and that most engineers are watching the wrong gauges.

---

The Building Blocks: What Actually Handles a Connection

Before getting to failure modes, it helps to have a concrete model of what happens when a user connects to a server.

File Descriptors

The OS tracks every open resource — files, network connections, pipes — using a small integer called a file descriptor (FD). The first three are always reserved:

FD 0 → stdin  (keyboard input)
FD 1 → stdout (console output)
FD 2 → stderr (error output)
FD 3 → your log file
FD 4 → user A's connection
FD 5 → user B's connection
...
FD 1024 → OS says "EMFILE — no more" ❌

Every TCP connection your server accepts consumes one FD. The default per-process limit on Linux is 1,024. When that fills, the OS returns EMFILE (too many open files) and refuses new connections — regardless of how much CPU or memory you have left.

$ ulimit -n        # check current limit → probably 1024
$ ulimit -n 65535  # raise it (current session only)
# Permanent: edit /etc/security/limits.conf

TCP Connections: The Handshake

Every HTTP connection starts with a TCP three-way handshake before any data flows:

Client → Server: SYN      ("I'd like to connect")
Server → Client: SYN-ACK  ("Got it, I'm ready")
Client → Server: ACK       ("Confirmed — send your request")
Client → Server: [data]    (the actual HTTP request)

Each step takes a network round-trip. For a client in the same data center that's ~0.1ms; across continents it's 100–300ms. This overhead motivated HTTP Keep-Alive.

Keep-Alive: Efficiency With a Hidden Cost

HTTP Keep-Alive reuses a single TCP connection for multiple requests instead of handshaking for each one:

Without: handshake → request → close,  handshake → request → close  (2 handshakes)
With:    handshake → request → request → request → close             (1 handshake)

Faster for the client. But idle keep-alive connections still hold an FD open on the server doing nothing. 5,000 users sitting on an idle browser tab = 5,000 FDs consumed. This is a trap that ended the startup in the war story below.

TIME_WAIT: The Port Cooldown

When a TCP connection closes, the port on the client side enters TIME_WAIT for 60–120 seconds. During this window, that port can't be reused for new outbound connections.

1,000 outbound connections/sec × 60 sec TIME_WAIT = 60,000 ports locked
Total available ephemeral ports: ~65,000
→ Port exhaustion, even with healthy CPU and memory

This matters most on servers that make many outbound connections (proxies, services that call many APIs). You'll see connect: Cannot assign requested address in logs.

---

The Resource Landscape

A server has five independent ceilings, each with its own failure signature:

Resource	What fills it	Failure signature	Default limit
File descriptors	TCP connections, open files	EMFILE — new connections refused	1,024/process
Memory	Heap, thread stacks, buffers	OOM Killer silently kills process	Depends on RAM
CPU	Computation, context switching	All requests slow down together	100% across cores
Disk IOPS	Read/write operations	Write latency spikes	SSD: ~100K–500K/s
Network ports	Outbound connections (TIME_WAIT)	New outbound connections fail	~65K ephemeral ports

The critical distinction: CPU and memory saturation cause gradual degradation — everything slows down proportionally. FD and port exhaustion cause total failure — new connections are refused while existing ones and dashboard metrics look completely healthy. The second kind is the one that produces a 47-minute outage where nobody can figure out why the server is "down."

---

Concurrency Models: How the Server Handles Multiple Users

How a server handles multiple simultaneous connections determines which ceiling it hits first. There are three main approaches.

Thread-per-Connection

A new OS thread is created for each incoming connection. That thread handles everything for that connection — reading the request, processing it, writing the response — then terminates.

User A connects → Thread 1 created (1MB RAM)
User B connects → Thread 2 created (1MB RAM)
User C connects → Thread 3 created (1MB RAM)
...
User 32,000 → Thread 32,000 → OOM

Simple to code: each thread runs straight-line blocking code. But each thread costs ~1MB of stack space, and context switching between thousands of threads burns CPU. At 10,000 threads, 10–30% of CPU goes just to saving and restoring thread state. At 100,000 — the system is unusable.

First ceiling: Memory (32GB ÷ 1MB/thread ≈ 32,000 max threads), then CPU from context switching.

Event Loop

One thread handles all connections by processing callbacks as they become ready. Instead of blocking while waiting for I/O, it registers a callback and moves on.

Hotel receptionist analogy:
  Loop forever:
    1. Any new guests?        → start check-in
    2. Any responses ready?   → send them
    3. Any room service done? → notify the room (non-blocking)
    4. Back to step 1

ONE receptionist. MANY guests. Never stands idle waiting.

Each connection costs only a few KB (just a file descriptor and some socket state), not 1MB. The same 32GB server can hold 100,000+ connections.

First ceiling: File descriptors (OS limit), or CPU if any one request does synchronous computation.

Weakness: Head-of-line blocking. One slow synchronous task holds the single thread, freezing every other connection behind it.

Drive-through analogy:
  Car A: "Custom birthday cake" (10 min)
  Car B: "Just a coffee"  ← waits 10 minutes behind Car A
  Car C: "A muffin"       ← waits 10:30

Thread Pool

A fixed number of pre-created threads pull work from a shared queue. Balances the two extremes: bounded memory cost (N threads, not one per connection), and true parallelism across CPU cores.

Hire 20 waiters. All new tables wait in the lobby.
A free waiter → grabs next table → serves → returns to lobby.
If all 20 are busy → table waits in queue.
If queue is full → reject with 503.

The pool size formula:

pool_size = num_cores × (1 + wait_time / compute_time)
Example: 8 cores, 90ms DB wait, 10ms compute → 8 × 10 = 80 threads

Failure mode: If a slow downstream dependency holds all threads (e.g., 200 threads × 30-second timeout = all blocked), the queue fills and the service rejects everything — even though your code is fine.

The Comparison That Matters

	Thread-per-Connection	Event Loop	Thread Pool
Memory @ 10K connections	~10 GB	~50 MB	~500 MB
Max practical connections	1K–10K	10K–100K+	10K–50K
Failure mode	OOM / context-switch death spiral	Head-of-line blocking	Queue overflow
Race conditions	Every CPU instruction boundary	Only at await boundaries	Every CPU instruction boundary
Who uses it	Apache httpd	Node.js, nginx, Redis	Java Tomcat, Go/Netty

The "Pure Event Loop" Lie

Node.js is described as single-threaded. It isn't, exactly. The event loop handles network I/O directly (non-blocking), but file I/O and DNS lookups are blocking operations that can't be done asynchronously by the OS. Node offloads these to a small internal thread pool via libuv:

Event loop handles:          libuv thread pool (default: 4 threads) handles:
  TCP connections              fs.readFile / writeFile
  HTTP parsing                 DNS lookups (dns.lookup)
  DB queries over network      crypto (pbkdf2, scrypt)
  setTimeout / setInterval     zlib compression

The 4-thread default is dangerously low in production. If 4 file reads are in progress, a DNS lookup queues behind them. HTTP calls to external APIs start timing out for no obvious reason. The fix: UV_THREADPOOL_SIZE=64 set before the process starts.

Every "event loop" runtime is actually a hybrid. Knowing which operations go to the thread pool and which stay on the event loop is essential for debugging latency spikes.

---

What the Experiments Showed

The prototype: a Node.js HTTP key-value store, raw http module, zero npm packages. Four routes: /data/:key (read/write), /stats (live metrics — memory, FDs, event loop lag), and /heavy (a deliberate CPU blocker). Simple enough to break in predictable ways.

Experiment 1 — Connection flood: the ceiling was the OS kernel

Opening TCP connections without closing them, stepping up in batches. The server stayed alive until:

15,965 connections → ENOBUFS (no buffer space available)
CPU: ~0%    Node heap: ~480 MB (healthy)

The event loop was not blocked. Node's memory was fine. What ran out was OS kernel network buffer space — roughly 250 MB of non-paged pool memory consumed by socket receive/send buffers (~16 KB each). The ceiling was in the kernel, not in the application.

Ceiling hierarchy hit in order:
  1. OS kernel network buffers  ← actual ceiling on Windows
  2. FD limit (~16K on Windows) ← would have been next
  3. Node.js heap               ← never reached
  4. CPU                        ← never reached

This is the war story that opens the phase: a social media API during a viral spike. CPU at 5%, memory at 40%, every dashboard looked healthy — but the server was refusing every new connection. 47-minute outage. The fix:

ulimit -n 65535   # raises the per-process FD limit from the default 1,024

The point isn't to memorize ulimit. It's that the server had massive headroom on four of its five ceilings, and failed on the fifth one nobody was measuring.

Experiment 2 — One CPU-bound request freezes all 15,965 users

A single request to /heavy:

// This holds the event loop thread — nothing else can run
const start = Date.now();
while (Date.now() - start < 5000) {}

During those 5 seconds:

/stats response time:  5,000 ms  (normally 1 ms)
Event loop lag:        12,000 ms
CPU:                   98%
All other requests:    frozen

The /stats route does trivial work — read process memory, count handles, respond. Under normal conditions it takes 1ms. The while loop on /heavy blocked it for 5 full seconds because the event loop is a single thread with cooperative scheduling. There is no OS scheduler interrupting a running synchronous task to let another request proceed. The CPU-bound route held the thread until it finished.

This surfaces in production from: JSON.parse() on a 50 MB payload, an unanchored regex on user input (ReDoS), or a forgotten fs.readFileSync() in a request handler. All of them stall every connected user simultaneously.

Experiment 3 — Async doesn't make reads safe

100 concurrent PUT requests, each reading the current counter, incrementing it, and writing it back. Expected final value: 100. Actual: 60-something.

The interleaving that causes this:

Request A: await readStore()   → gets { counter: 1 }  ← yields here
Request B: await readStore()   → gets { counter: 1 }  ← stale; A hasn't written
Request A: await writeStore({ counter: 2 })
Request B: await writeStore({ counter: 2 })            ← overwrites; should be 3

The assumption going in: single-threaded event loop means no interleaving. That's true between await calls — code between two awaits is atomic. But the full async function is not. Every await is a yield point where the event loop can run another callback. In a read-modify-write pattern, that's enough.

The same lost-update problem exists in multi-threaded code, just at every CPU instruction instead of only at await boundaries. The event loop makes races less frequent, not impossible.

---

The Fix Reveals the Lesson

Each experiment has a distinct fix:

• FD/buffer exhaustion → raise ulimit -n / set nofile in systemd unit or container config
• Event loop blocking → offload synchronous CPU work to worker_threads; never call blocking APIs in a hot path
• Async race conditions → serialize writes through a queue, or use atomic DB operations (transactions, compare-and-swap)

All three experiments produced similar surface symptoms — slow or unresponsive server. The root causes are completely different. Applying the wrong intervention (or adding a second server) wouldn't have helped. The resource that fails determines both the symptom and the remedy.

---

What This Phase Actually Teaches

Not all resource ceilings behave like slowness. FD and port exhaustion produce hard refusals while CPU and memory metrics look healthy. This class of failure is disproportionately misdiagnosed because teams are watching the wrong gauges.

The concurrency model determines which ceiling you hit first. Thread-per-connection hits memory first (~1MB/thread → ~32K threads on 32GB RAM). Event-loop hits file descriptors or CPU-bound blocking first. The model isn't just a performance choice — it changes the failure mode.

What This Phase Actually Teaches

1. A server has five independent resource ceilings, and the smallest one is the actual limit. CPU saturation causes gradual degradation. FD exhaustion and port exhaustion cause total failure while all other metrics look fine. Know which ceiling you're approaching before an incident.

2. The thread-per-connection model kills itself on memory and context switching. At 10,000 threads: ~10 GB of stack space, and 10–30% of CPU wasted on saving/restoring thread state. This is why Node.js, nginx, and Redis all chose the event loop.

3. The event loop trades memory efficiency for CPU vulnerability. A single synchronous task holds the thread. Head-of-line blocking isn't a bug to fix — it's a fundamental property of cooperative concurrency. The architecture choice is: which failure mode can I tolerate? OOM death spiral, or one hot request freezing everyone?

4. "Single-threaded" is a simplification. Node.js has 4 libuv worker threads by default handling file I/O, DNS, and crypto. DNS lookups queue behind file reads. In production, set UV_THREADPOOL_SIZE=64.

5. Async code has race conditions at await boundaries. Code between two awaits is atomic. A full async function is not. Every read-modify-write pattern across awaits needs explicit serialization.

When to use which model:

< 5K connections, simple code needed        → Thread-per-connection
> 10K connections, mostly I/O-bound         → Event loop
Mixed CPU + I/O, need parallelism           → Thread pool
Production at scale                         → Hybrid (event loop + worker pool)

---

Where I'm Still Fuzzy

• ENOBUFS vs EMFILE — I hit ENOBUFS at ~16K connections on Windows, which maps to OS kernel buffer exhaustion, not the FD limit. On Linux with default ulimit -n 1024, EMFILE would appear much earlier. Are these always separable in production?

• TIME_WAIT at high connection churn — I understand port exhaustion in theory (1K conns/sec × 60s = 60K locked ports), but I haven't built something that actually hits it. That would require a high-outbound-connection scenario — probably relevant in Phase 9 (service decomposition) when services call each other.

• libuv thread pool saturation in practice — the 4-thread default is clearly too low, but what does the performance curve look like as you raise it? Is there a point where more threads hurt?

---

Four experiments worth running:

1. GET /stats — see live FD count, heap usage, and event loop lag
2. Hit /heavy in one tab, then hit /stats from another — watch the lag number
3. Run 100 concurrent writes to a counter and check if the final value matches the expected count
4. Open thousands of connections without closing them and watch when the OS complains first

The stats endpoint measures event loop lag by scheduling a setTimeout(0) callback and measuring how long before it actually runs. Under normal load: < 5ms. During a CPU-bound request: in the thousands.

---

Next

Phase 2 moves the problem to multiple machines — which immediately surfaces a new class of failures that single-machine thinking doesn't predict.

---

Part of a 12-phase series building distributed systems intuition from first principles. Each phase has a running prototype, failure scenarios, and a gate check before moving on.