DEV Community: HappyBug

System Design the Agentic Way — Phase 3: Data Storage Trade-offs

HappyBug — Sun, 26 Jul 2026 11:18:31 +0000

Phase 3: Data Storage Trade-offs

Phase 2 leaned on a shared store and waved at durability — "the data survived a Redis restart" — then admitted the caveat: a graceful shutdown snapshotted to disk, an ungraceful one wouldn't have. Phase 3 goes inside that store and asks the question the caveat was hiding: when you tell a client saved, what exactly did you promise, and what does keeping that promise cost?

The Question
The Building Blocks: What "Saved" Actually Costs
- Where a Write Actually Goes: The Four Layers
- write() vs fsync and the fsync Tax
- Process Crash vs Power Loss
- The Durability Window
- The Write-Ahead Log
- Indexes and Write Amplification
- B-tree vs LSM: The Storage-Engine Fork
- The Trade-off Matrix
Durability Is Not Availability to Reads
The Fix Reveals the Lesson
What This Phase Actually Teaches
Where I'm Still Fuzzy
Try It Yourself
Next

The Question

Here's a question that sounds trivial until you try to answer it precisely: when your database replies 200 OK, where is your data at that exact moment? On the disk? In memory? Somewhere in between — a place a power cut would quietly erase a half-second later? Most engineers can't say, and the ones who found out the hard way have a war story about it.

Chase that question and it forks into three: How do I not lose data on a crash? How do I make writes fast? How do I make reads fast? Every storage engine on Earth is just a particular answer to those three — and they pull against each other, so no engine wins all three. The tension between them is the entire phase, and by the end you'll be able to answer the opening question for any system, including the one you build here.

                    DURABILITY
              (don't lose acked writes)
                        /\
                       /  \
                      / .  \        every storage engine
                     /  .   \       lives somewhere inside
                    /   .    \      this triangle — push
                   /    .     \     toward one corner and
                  / .   .    . \    you give up the others
                 /______.______\
          WRITE SPEED         READ SPEED
        (cheap appends)     (indexes, sorted)

A war story sets the stakes. An analytics SaaS, Series B, ran an ingestion service that accepted event writes, buffered them in memory, and flushed to disk every 200 ms in a batch — because "fewer, bigger writes are faster." The client got a 200 OK the instant the write hit the in-memory buffer. Everyone was proud of the p99.

Then a routine node recycle during autoscaling sent SIGKILL to a pod holding ~200 ms of un-flushed writes. At 3:47 AM no alert fired; the service came back healthy. Days later a customer proved from their own logs that ~4,000 events they'd received a 200 OK for simply weren't in any report. The team blamed the query layer, then the cache, then a "customer clock issue." The truth was simpler: the writes were acknowledged before they were durable. A crash inside the 200 ms window was invisible data loss — no error at write time, no error at read time, just missing records.

The lesson is a single question — when you tell a client "saved," what have you actually promised? — and the rest of this article is what it takes to make that promise honest.

The Building Blocks: What "Saved" Actually Costs

Where a Write Actually Goes: The Four Layers

A write() does not put data on disk. It passes through four layers, and only the last one is non-volatile:

your string
   │  write()               microseconds
   ▼
① App buffer      (process RAM)          dies on power loss AND on process crash
   ▼
② OS page cache   (kernel RAM, dirty)    dies on power loss — SURVIVES a process crash
   ▼
③ Disk controller cache (RAM on the SSD) dies on power loss
   ▼
④ NAND / platter  (non-volatile media)   survives everything  ← the only durable layer

Layer	Location	Volatile?
① App buffer	Process RAM	dies on power loss (and process crash)
② OS page cache	Kernel RAM (dirty pages)	dies on power loss
③ Disk controller cache	RAM on the SSD	dies on power loss
④ NAND / platter	Non-volatile media	survives ← durable

NAND is the array of non-volatile flash chips inside an SSD (the HDD equivalent is a magnetic platter). Getting bytes from layer ① to layer ④ is the whole job. Everything above ④ is a form of RAM that is lying to you about durability until proven otherwise.

write() vs fsync and the fsync Tax

Two syscalls, two very different guarantees:

write() returns → your bytes are in the OS page cache (②). They survive a process crash, because the kernel outlives the process. They do not survive power loss.
fsync(fd) returns → the kernel has flushed those pages down to NAND (④) and blocked until the disk confirmed. Now the write survives everything.

The gap between them is the price of durability:

write()   ≈ microseconds        (just a memcpy into kernel RAM)
fsync()   ≈ 0.1–2 ms  on SSD    (100–1000× slower)
          ≈ 5–15 ms  on HDD

That 100–1000× gap is the fsync tax. Skipping fsync makes writes look dramatically faster — which is exactly why teams skip it, and exactly how they end up acking data they haven't actually saved.

Process Crash vs Power Loss

This is the crux the analytics team missed, so it's worth being precise. The two failure classes have opposite consequences for un-fsync'd data:

Process crash  → kill -9 / OOM / deploy / pod eviction
                 kernel survives → page cache (②) survives → un-fsync'd data ✅ SURVIVES

Power loss     → power event / --force stop / spot reclaim / host or kernel failure
                 RAM dies → page cache (②) is gone → un-fsync'd data ❌ LOST

The trap is subtle: a process crash — kill -9, an OOM, a deploy — leaves the page cache alive, so un-fsync'd data quietly survives it. Only a power-class event — a real power loss, a kernel panic, a forced host stop — erases that RAM. Durability has to hold against the second class, and the happy path never exercises it, which is why the loss tends to surface weeks later in someone else's logs.

The Durability Window

If you fsync on a timer instead of on every write, there is a slice of acknowledged data that isn't yet on disk — the durability window:

fsync policy	Window of exposure
`appendfsync no`	seconds (whenever the OS feels like flushing)
`appendfsync everysec`	≈ 1 s — Redis's default; the fsync timer runs independent of your write
`appendfsync always`	≈ 0 — one fsync per write, maximum safety, maximum tax

The subtle mistake here: "fsync every second" is not "my write is synced within a second of my write." The timer fires on its own schedule; a crash can land in the instant right before the next tick, taking a nearly-full second of acked writes with it. The window is a property of the timer, not of any individual request.

The Write-Ahead Log

The fix for all of this is the Write-Ahead Log (WAL): write the change to a durable, append-only log first — ahead of applying it to the real data structure — and only ack once that log entry is on disk. Log first, apply second.

SET balance=100
 ① append to WAL      (sequential, at the end of the file)
 ② fsync(wal_fd)      (one cheap flush → durable)
 ③ return 200 OK      (now it is safe to say "saved")
 ...later, in the background...
 ④ apply to the real data structure

Why a log instead of just updating the data structure in place?

Append is cheap because it's sequential. The write always goes to the end of one file — no seeking. A B-tree update, by contrast, touches scattered pages all over the disk; fsyncing those is slow and a crash mid-update leaves a half-written structure = garbage.
Recovery is a replay. On restart you read the log and re-apply entries in order. An incomplete tail entry (a crash mid-append) is detected by length/checksum and discarded — it was never acked, so dropping it is safe.
Never write in the middle. The log is append-only on purpose: it preserves order (the history is the source of truth), keeps writes sequential, and keeps already-durable bytes immutable so only the final entry can ever be torn. You reclaim space with a checkpoint/truncation or a compaction that writes a fresh file and deletes the old one — never an in-place edit.

An analogy makes the ordering guarantee concrete:

Analogy:   a bank ledger — you never erase an old line; you append a correcting entry.
Technical: SET x=1 then SET x=2 both append. On power loss, replay in order → GET x = 2.
           Last write wins because log order is the source of truth.

This pattern is not a toy — it is everywhere:

System	Its WAL
PostgreSQL	the WAL (`pg_wal`)
SQLite	WAL journal mode
RocksDB / LevelDB	the write-ahead log in front of the memtable
Kafka	the entire log is the database
Redis	AOF (Append-Only File) — the AOF is a WAL; `appendfsync always` = window ≈ 0

One optimization worth naming: group commit. Doing one fsync per write caps throughput at N fsyncs/sec. Instead, batch every write that arrives in a 1–10 ms window, append them all, do one fsync for the batch, then ack all of them. Each client waits a hair longer; total throughput jumps, because the expensive syscall is amortized across the batch.

Indexes and Write Amplification

An append-only log is wonderful to write and terrible to read: finding a key means a full scan — read every record, O(n).

An index is a second, sorted data structure that answers "where is key X?", turning that O(n) scan into O(log n) or O(1). It's a derived copy kept in sync with the data — like the index at the back of a textbook. It is not the source of truth.

But indexes aren't free, and the cost has a name: write amplification — the ratio of physical writes to logical writes. Every index has to be updated on every write, so five indexes mean roughly five times the disk I/O per logical write.

Analogy:   the "Index Everything" reflex — index every column "just in case".
Technical: 5 indexes ≈ 5× write I/O. The signal you fell for it:
           your writes are slower than your reads.

An index is a trade — read speed bought with write cost and disk space — not a free win. That single reframing is what separates "add an index" as a reflex from "add an index" as a decision.

B-tree vs LSM: The Storage-Engine Fork

Once you accept that write amplification is real, the whole storage-engine design splits into two families based on how they lay writes down:

  B-tree — UPDATE IN PLACE              LSM — APPEND, MERGE LATER

  write ─┐                              write ─┐
         ▼                                     ▼
   seek to the right page              memtable  (RAM, sorted)
         ▼                                     │ fills up
   modify it where it lives                    ▼
         ▼                              flush ─► SSTable  (immutable file)
   maybe split / rebalance                     │  SSTable
                                               │  SSTable
  → random I/O, scattered writes               ▼
  → great reads (one lookup)          compaction merges them
                                       → sequential I/O, never in place
                                       → high write throughput

	B-tree (Postgres/MySQL)	LSM (Cassandra/RocksDB)
Write path	random, in-place	sequential append
Write throughput	moderate	high
Read path	one lookup down the tree	memtable + several SSTables
Read amplification	low (~1)	higher (N files)
Background work	page splits / rebalancing	compaction (CPU/I/O bursts)
Best when	balanced/read-heavy, rich queries + txns	write-heavy (logs, metrics, events)

An LSM tree is, in effect, "the WAL idea applied to the whole database." Its parts:

memtable   → in-RAM sorted table; every write lands here first (fast, sequential)
SSTable    → Sorted String Table: an immutable sorted file flushed from the memtable
compaction → background merge of SSTables; drops overwritten/deleted keys,
             writes new files and deletes the old ones — never in-place

The decision falls out of the workload. For a metrics system doing millions of writes/second where reads are mostly for recent data, you choose LSM: writes become cheap sequential appends, and recent reads hit the memtable and the newest SSTables so read amplification barely bites. A B-tree is ruled out at that write rate by random-I/O seeks and write amplification from page rewrites and splits. This is exactly why Cassandra, ScyllaDB, and InfluxDB are LSM-based.

The Trade-off Matrix

Widen the lens to include the append-only log as a first-class storage choice, and the full picture is three columns:

	B-tree (Postgres)	LSM (Cassandra)	Append-only log (Kafka)
Write throughput	moderate	high	highest
Point lookup	excellent	good	poor
Range/sorted scan	excellent	good	poor
Write amplification	page rewrites + splits	compaction (amortized)	~none
Read amplification	~1	N (worst on old data)	N/A
Operational cost	mature, predictable	compaction tuning	consumers track offsets
Reach for it when	balanced/read-heavy + txns	write-heavy, key/time-series	event streams, ordered replay

How to read it: don't pick the "best" engine — pick the column whose strength matches your bottleneck and whose weakness you can tolerate. Real systems mix engines: Postgres for accounts, Cassandra for the firehose, Kafka to move between them. "One database for everything" is the trap.

Durability Is Not Availability to Reads

The prototype is a durable key-value store: an append-only wal.log as the source of truth, an in-memory Map as a derived read cache rebuilt on startup. The write path is log-first, and the durability contract lives in one function — the ack happens only after the fsync returns:

function appendToWAL(logEntry) {
  const line = JSON.stringify(logEntry) + "\n";

  // Reaches only the OS page cache (layer ②) so far.
  fs.writeSync(fd, line);

  // Force page cache → NAND (④), and BLOCK until the disk confirms.
  fs.fsyncSync(fd);

  // We return only AFTER the fsync. The caller can now honestly say "OK".
}

Building the write path first exposed a gap that the phrase "the data survived" always hides. After appending two writes and restarting the server, GET x returned 404 — while wal.log still held both entries on disk.

Nothing was lost. The WAL faithfully kept every write across the restart. But the thing that answers a GET is the in-memory Map, and a fresh process starts with an empty one (in-memory keys at startup: 0). The history was recorded and never replayed. Durability is not availability to reads — they are two independent properties, and the WAL pattern has two halves to match: append on write (done) and replay on recovery (missing).

  AFTER A RESTART, BEFORE recovery runs:

    wal.log (disk)   │  x=1   y=hello        ← durable, intact, still on disk
    Map    (memory)  │  (empty)              ← GET x → 404
                     └────────────────────────────────────────────
                        the data IS saved, but nothing can read it

  recover()  replays the log back into memory:

    wal.log ──► parse each line in order ──► apply() ──► Map
      x=1                                                x=1
      y=hello                                            y=hello
                     └────────────────────────────────────────────
                        now GET x → 200 OK (value 1). gap closed.

The second half is recover(), called before the server accepts any request:

function recover() {
  const raw = fs.readFileSync(WAL_PATH, "utf8");
  const lines = raw.split("\n");        // clean file ends in "\n" → last element is ""
  let applied = 0, discardedTail = false;

  for (let i = 0; i < lines.length; i++) {
    const line = lines[i];
    if (line === "") continue;
    try {
      apply(JSON.parse(line));          // same apply() the live write path uses
      applied++;
    } catch (err) {
      // Only the LAST line may be broken — a torn tail append that was never acked.
      if (i === lines.length - 1) discardedTail = true;   // safe to drop
      else throw new Error(`WAL corruption at line ${i + 1}`); // real corruption → fail loud
    }
  }
  return { applied, discardedTail };
}

Two details carry the design. First, apply() is shared by both the live set/del path and replay, which guarantees a write and its recovery are interpreted identically. Second, the torn-tail rule is asymmetric on purpose: a parse error on the last non-empty line is a half-written append (never acked) and is discarded; a parse error anywhere else is real corruption and throws loudly.

With recovery in place, the same fresh process that returned 404 now serves GET x = 1 (HTTP 200) — recovered 2 WAL entries → 2 keys. And last-write-wins holds across a restart: PUT x=1 then PUT x=2 appends a new line rather than editing in place, so a restart replays recovered 3 WAL entries → 2 keys and GET x = 2. Replay order is write order.

The Fix Reveals the Lesson

Every failure in this phase traces to the same root — a layer of RAM being mistaken for durable storage — and each has a matching fix:

Ack before durable → append to the WAL and fsync before returning 200 OK. The ack is only honest once the fsync confirms.
Data on disk but reads 404 → replay the WAL into the read structure on startup. Durability and read-availability are separate; you must build both halves.
Buffering in app memory → get bytes out of layer ① as early as possible; a per-request flush trades latency for the elimination of a silent-loss window (or use group commit to claw the latency back).

The shape is the same as Phases 1 and 2: the component that fails determines both the symptom and the fix. What's new is that the failure is invisible at both write time and read time — no error is ever returned. It surfaces only under a power-class failure that erases RAM, which the happy path never exercises — so the loss can reach production before anyone notices.

What This Phase Actually Teaches

1. A write() is not durable; only fsync is. Bytes pass through four layers and only NAND survives power loss. Until fsync returns, "saved" is a claim about RAM, not disk.

2. Process crash and power loss are different failure classes. Un-fsync'd data lives in the OS page cache — kernel memory — so it survives a process crash but dies on power loss. Conflating the two is how silent loss reaches production.

3. A WAL makes "saved" honest, cheaply. Append-first + fsync-before-ack buys durability with sequential I/O; recovery is an in-order replay; only the untorn, acked prefix is ever restored. It's the pattern under Postgres, Kafka, SQLite, and Redis AOF.

4. Durability is not availability to reads. The log surviving a crash and the reads working after a crash are independent guarantees. You have to build append-on-write and replay-on-recovery.

5. Every index is a trade. Read speed is bought with write amplification and space. "Index everything" shows up as writes slower than reads. B-tree vs LSM is the same trade at engine scale: pick the write/read amplification profile your workload can tolerate.

Decision heuristic:

Must never lose an acked write            → WAL: append + fsync BEFORE ack
Throughput-bound by fsyncs/sec            → group commit (batch, one fsync, ack all)
Can tolerate a ~1s loss window for speed  → fsync on a timer (everysec), eyes open
Balanced/read-heavy + rich queries + txns → B-tree (Postgres/MySQL)
Write-heavy, key or time-series, recent reads → LSM (Cassandra/RocksDB/Influx)
Ordered event stream, replayable          → append-only log (Kafka)
Reads slower than a full scan can afford   → add ONE index, and only the ones you query

Where I'm Still Fuzzy

Where the durability window should actually sit. everysec vs always is a latency-vs-loss dial, but I don't have a principled way to choose the tolerance. Presumably it's a function of how much acked data the business can afford to lose per crash — but I haven't turned that into a number I'd defend.
Torn-write detection beyond a parse error. Discarding a JSON line that won't parse catches an obviously truncated tail, but a torn write could in theory leave a line that does parse yet is semantically wrong. Real engines use checksums per record. I understand why, but I haven't built the checksum path to feel where the naive length/parse check breaks.
When group commit's added latency is unacceptable. Batching amortizes fsync beautifully for throughput, but it deliberately makes each individual write wait. For a low-traffic, latency-sensitive write path the batch window might be pure overhead. I don't yet know where the crossover is.

Try It Yourself

The prototype is at github.com/aishwarya-chamanoor/system-design-agenticway in phase-03/.

git clone https://github.com/aishwarya-chamanoor/system-design-agenticway
cd system-design-agenticway/phase-03
node server.js

Experiments worth running:

PUT /kv/x a value, restart the server, then GET /kv/x — watch the startup log say recovered N WAL entries and the read return your value. That's replay-on-recovery working.
PUT the same key twice with different values, restart, GET — the second value wins, because the WAL appended a new line instead of editing in place. Observe last-write-wins falling out of log order.
Peek at wal.log between writes (it's newline-delimited JSON) and confirm it only ever grows — no line is ever rewritten. That append-only shape is the whole safety argument.

Phase 4 asks what happens when reads outrun the durable store this phase built — and what it costs to keep a faster copy of the truth without letting it drift.

Part of a 12-phase series building distributed systems intuition from first principles. Each phase has a running prototype, failure scenarios, and a gate check before moving on.

System Design the Agentic Way — Phase 2: Stateless Horizontal Scaling

HappyBug — Tue, 14 Jul 2026 11:09:21 +0000

Phase 2: Stateless Horizontal Scaling

Phase 1 ended at a wall: a single machine has hard ceilings, and no amount of better code moves them. The obvious answer is "add more servers." Phase 2 is about everything that sentence quietly assumes — and the new failure modes that appear the moment a second machine exists.

The Question
The Building Blocks: What "Add a Server" Actually Requires
- State: The Thing That Breaks First
- The Shared-Nothing Principle
- Sticky Sessions: The Tempting Trap
- Load Balancing Algorithms
- Layer 4 vs Layer 7
- Latency vs Throughput
Concurrent Requests Don't Run Sequentially
What the Experiments Showed
The Fix Reveals the Lesson
What This Phase Actually Teaches
Where I'm Still Fuzzy
Try It Yourself
Next

The Question

You have one server and you want three. What has to be true about your application before adding the other two actually helps — and where does the bottleneck go once you do?

Phase 2 is an argument that "add more servers" is not an infrastructure task. It's an application-architecture constraint that decides whether the extra machines do anything at all.

The Building Blocks: What "Add a Server" Actually Requires

A war story sets the stakes. A food delivery startup ran its entire API on one server. A routine Friday 6pm deploy meant an 8-second restart. During those 8 seconds: 47 lost payment confirmations, 312 connection resets, and — because the process had to reload a 2.3 GB data.json — a 45-second full recovery. The bill was $23,000 in refunds.

The root cause isn't the deploy. It's that a single machine can't be restarted without downtime and can't survive its own hardware dying. The fix is to run several identical copies. But copies only work if the servers hold no state — and that word hides most of the difficulty.

State: The Thing That Breaks First

Take the Phase 1 server and imagine running three copies of it. Three distinct kinds of state each break in their own way:

Type of state	Example	Why 3 copies break it
Persistent data	`data.json` on disk	Each server has its own copy → writes diverge, reads disagree
In-memory counters	`activeConnections`	Each server sees only its own slice → no global view
Coordination state	write queue / locks	Only valid inside one process → no cross-server ordering

The pattern: anything a server remembers between requests becomes wrong the moment a second server exists, because the next request might land somewhere else.

The Shared-Nothing Principle

The fix is to externalize all state to a shared service — a database, a cache like Redis, a metrics store like Prometheus — so that no server owns anything. The server becomes a pure function:

request in → (read shared state) → (write shared state) → response out
server remembers NOTHING between requests

This is the shared-nothing architecture. Any request can go to any server, because every server is interchangeable. That interchangeability is the entire point — it's what makes servers disposable, and disposable servers are what let you deploy, restart, and lose hardware without an outage.

Sticky Sessions: The Tempting Trap

If a server does hold per-user state (a login session in memory), you can paper over it by pinning each user to the same server every time — a sticky session.

Stateful  + sticky sessions → user X always routed to Server 2
Stateless + any routing     → user X can be served by any server

The trap: sticky sessions reintroduce the single-machine failure. When Server 2 dies, every user pinned to it loses their session — logged out, cart emptied — even though two other servers are healthy. Sticky sessions are a temporary crutch. The goal is always full statelessness, where a dying server costs nothing.

A related temptation is to jump straight to microservices "to reduce blast radius." Scale the monolith horizontally first: three identical copies, deployed one at a time (a rolling deploy), gives zero-downtime with zero architectural change.

Monolith, 1 server      → single point of failure
Monolith, 3 servers     → scales, survives restarts, still SIMPLE
Microservices, 20 svcs  → scales, but adds network failures + distributed transactions

Decompose only when there's a specific reason to (that's Phase 9), not reflexively.

Load Balancing Algorithms

With three interchangeable servers, something has to decide who gets each request. That's the load balancer, and the decision rule matters:

Algorithm	How it works	Best for	Weakness
Round Robin	A, B, C, A, B, C…	Equal servers, equal-cost requests	Ignores how heavy each request is
Least Connections	Route to server with fewest active connections	Mixed request durations	Blind to raw server capacity
Weighted Round Robin	More traffic to beefier servers	Heterogeneous hardware	Static, doesn't adapt to load
Random	Pick any server	Surprisingly good at scale	Can cluster at low request counts

For a server where /heavy takes 5 s but /data/:key takes 5 ms, Least Connections is the right pick. Round Robin would keep handing new requests to a server already stuck on a slow one:

Round Robin:  Server A busy on /heavy (5s) → still gets 2 more → queue builds
Least Conn:   Server A shows 3 active     → skip it → route to Server B (0 active) ✓

Layer 4 vs Layer 7

"Reverse proxy" and "load balancer" are not separate things — any proxy that hides backends can trivially point at several of them. The meaningful distinction is which network layer it operates on:

Layer 4 (transport):  routes packets by IP + port. Fast, protocol-blind.
                      Example: AWS Network Load Balancer.
Layer 7 (application): understands HTTP — can route by path, header, cookie.
                      Example: Nginx, HAProxy, Envoy.

Layer 7 costs a little more per request but can make smart, content-aware routing decisions. This prototype uses Nginx (Layer 7).

Latency vs Throughput

Externalizing state has a cost, and naming it correctly matters. Two independent measures:

Latency    = how long ONE request takes         (a single car driving A → B)
Throughput = how many requests complete per unit (cars arriving per hour)

Adding servers (or highway lanes) does not make a single request faster. It lets the system handle more requests at once. Concretely, moving from a local file to an external store:

	Phase 1 (local file)	Phase 2 (external store)
Latency	~0.2 ms	~5 ms (25× slower)
Throughput	~5K req/s	~30K req/s (6× more)
Failure tolerance	zero	survives a server dying

The trade is deliberate: accept higher per-request latency in exchange for far higher throughput and the elimination of the single point of failure. As long as latency stays under the ~100 ms human-perception threshold, users don't notice — but they very much notice an outage.

Concurrent Requests Don't Run Sequentially

A common misread: "if each request now waits 5 ms on the database, 10,000 requests take 50 seconds." That's wrong, and understanding why is the crux of the phase.

An await yields the event loop. While one request waits on the database, the server starts the next:

t=0.00 ms  Req 1 arrives → await db.read() → YIELDS
t=0.01 ms  Req 2 arrives → await db.read() → YIELDS
t=0.02 ms  Req 3 arrives → await db.read() → YIELDS
t=2.00 ms  Req 1's response comes back → resume → reply
t=2.01 ms  Req 2's response comes back → resume → reply

Each request individually experiences 5 ms. They overlap in time. The system does not fall over from per-request latency — it falls over when the database saturates:

3 servers × 10K req/s × 2 DB ops each = 60,000 DB ops/second

And that is the real lesson of horizontal scaling:

BEFORE:  [Server] ← ceiling was here (Phase 1: ~15K connections)

AFTER:   [Server 1] ─┐
         [Server 2] ─┼─→ [Database] ← ceiling moved HERE
         [Server 3] ─┘

Scaling doesn't eliminate bottlenecks. It moves them downstream.

What the Experiments Showed

The prototype makes this concrete: the Phase 1 server rewritten as an "amnesia server" (all fs calls replaced with Redis get/set), packaged into one Docker image, run as three containers behind an Nginx load balancer, all sharing one Redis.

Client → Nginx (:80) → server1:3000 ─┐
                       server2:3000 ─┼─→ Redis (:6379)  ← shared state
                       server3:3000 ─┘

Every response includes served_by: os.hostname() to prove distribution. Three verifications passed first:

Load distribution — 10 parallel /stats requests returned 3 different hostnames (a 4-3-3 split).
Shared state — a PUT through one server was immediately readable through the other two. Redis is the single source of truth.
Server death — docker stop server1, then read: the remaining two servers served all data with zero loss. The server is genuinely disposable.

Then came breaking it on purpose.

Experiment 1 — Kill the shared state: a hang is worse than a crash

docker stop on Redis, then a request. The expectation was an error response. What actually happened:

curl → server → await redis.get() → (never resolves) → request HANGS

No crash, no error — the Redis client silently retries the connection forever, so await redis.get() never settles. This is the worst failure mode: a crash returns an error instantly, but a hang ties up the connection, and enough hangs exhaust Nginx's connection pool and freeze the entire system. Always set timeouts on external service calls. A fast failure is recoverable; a silent hang cascades.

Experiment 2 — Restart Redis: did the data survive?

docker start on Redis, then read the old keys. The data was still there. Redis writes an RDB snapshot to disk on a graceful shutdown (SIGTERM).

The caveat matters: a docker kill (SIGKILL) or a real crash skips the snapshot, so everything since the last auto-save is lost. Graceful shutdown preserved durability here; an ungraceful one wouldn't have. Real durability guarantees are a Phase 3 topic.

Experiment 3 — Kill the load balancer: the new single point of failure

docker stop on Nginx, then curl localhost: connection refused. The servers and Redis were all healthy — reachable directly via docker exec — but unreachable from outside, because Nginx is the only public entry point.

Client → [Nginx]  → Server 1 → [Redis]
          SPOF       Server 2    SPOF
                     Server 3
                    (redundant)

Only the servers gained redundancy. Scaling moved the single point of failure (SPOF) from the server to two new places: the load balancer in front and the shared state behind.

The Fix Reveals the Lesson

Each break has a distinct remedy, and each remedy points at a later phase:

State divergence across servers → externalize every kind of state to a shared service; keep the server a pure function.
Hang on dependency failure → wrap every external call in a timeout (and eventually a circuit breaker — Phase 11).
Load balancer / shared-state SPOF → run multiple load balancers behind a cloud LB, and replicate the shared state (Phase 6).

The surface symptoms differ, but the shape is identical to Phase 1: the component that fails determines both the symptom and the fix. What changed is that "the component" is now something downstream of the server you added — which is exactly what horizontal scaling does to bottlenecks.

What This Phase Actually Teaches

1. Horizontal scaling is an application property, not an infrastructure one. Adding servers only helps if the servers are stateless. The hard work is externalizing state (data, counters, coordination); provisioning the machines is the easy part.

2. Shared-nothing makes servers disposable, and disposable is the whole goal. When any request can go to any server, you can deploy one at a time, tolerate hardware death, and restart freely — the three things a single machine can never do.

3. Sticky sessions quietly reintroduce single-machine failure. Pinning users to servers is a crutch; a dying server takes its pinned users down with it. Full statelessness is the target.

4. The load-balancing algorithm must match request-cost variance. With uniform requests, Round Robin is fine. With a mix of 5 ms and 5 s requests, Least Connections avoids piling work onto an already-busy server.

5. Scaling moves bottlenecks downstream — it never deletes them. Three servers turn one ceiling (connections) into a new one (database throughput at ~60K ops/s) plus two new SPOFs (the LB and the shared state). Know where the ceiling went.

6. A hang is worse than a crash. An unbounded wait on a dead dependency ties up resources and cascades. Every external call needs a timeout.

Decision heuristic:

Need zero-downtime deploys / HA, low complexity   → monolith × N behind a load balancer
Requests vary wildly in cost                      → Least Connections
Uniform requests, equal servers                   → Round Robin
Need path/header/cookie-aware routing             → Layer 7 (Nginx/Envoy)
Need raw packet throughput, protocol-blind        → Layer 4 (cloud NLB)
Independent scaling / team boundaries force it     → microservices (Phase 9), not before

Where I'm Still Fuzzy

Timeout values on external calls. I know an unbounded wait is the failure — but what's the right timeout? Too low and you fail healthy-but-slow requests; too high and you hang. It presumably depends on the dependency's p99, but I don't have a principled way to set it yet.
Load balancer redundancy in practice. "Run multiple Nginx behind a cloud LB" is the stock answer, but that just pushes the SPOF up to the cloud LB. Where does the regress actually stop — DNS? Anycast? I haven't built the layer above Nginx.
When Least Connections misleads. It's blind to server capacity, so a weak server with few connections still looks attractive. In a heterogeneous fleet, when does that go wrong, and is weighted-least-connections the real answer?

Try It Yourself

The prototype is at github.com/aishwarya-chamanoor/system-design-agenticway in phase-02/.

git clone https://github.com/aishwarya-chamanoor/system-design-agenticway
cd system-design-agenticway/phase-02
docker compose up

Four experiments worth running:

Hit http://localhost/stats ten times in parallel — watch served_by cycle across three hostnames.
PUT a value through one request, then GET it repeatedly — every server returns the same value because Redis owns it.
docker stop <server-container> and keep reading — the surviving servers serve everything; the server is disposable.
docker stop <redis-container> and send one request — watch it hang instead of erroring. That hang is the lesson.

Phase 3 goes inside the shared store this phase leaned on — asking what "the data survived" really guarantees, and what it costs to make that guarantee real.

Part of a 12-phase series building distributed systems intuition from first principles. Each phase has a running prototype, failure scenarios, and a gate check before moving on.

System Design the Agentic Way — Phase 1: The Single Machine Ceiling

HappyBug — Mon, 13 Jul 2026 13:34:05 +0000

Phase 1: The Single Machine Ceiling

Before reasoning about distributed systems, you need a concrete model of what a single machine actually runs out of. Phase 1 builds that model from the ground up — not from diagrams, but from pushing a server until it stops accepting connections, and understanding exactly what gave out and why.

The Question
The Building Blocks: What Actually Handles a Connection
- File Descriptors
- TCP Connections: The Handshake
- Keep-Alive: Efficiency With a Hidden Cost
- TIME_WAIT: The Port Cooldown
The Resource Landscape
Concurrency Models: How the Server Handles Multiple Users
- Thread-per-Connection
- Event Loop
- Thread Pool
- The Comparison That Matters
- The "Pure Event Loop" Lie
What the Experiments Showed
The Fix Reveals the Lesson
What This Phase Actually Teaches
Where I'm Still Fuzzy
Try It Yourself
Next

The Question

A server has CPU, memory, disk, and network capacity. When it fails under load, which one ran out — and does the answer change what you do about it?

Phase 1 is an argument that it matters enormously, and that most engineers are watching the wrong gauges.

The Building Blocks: What Actually Handles a Connection

Before getting to failure modes, it helps to have a concrete model of what happens when a user connects to a server.

File Descriptors

The OS tracks every open resource — files, network connections, pipes — using a small integer called a file descriptor (FD). The first three are always reserved:

FD 0 → stdin  (keyboard input)
FD 1 → stdout (console output)
FD 2 → stderr (error output)
FD 3 → your log file
FD 4 → user A's connection
FD 5 → user B's connection
...
FD 1024 → OS says "EMFILE — no more" ❌

Every TCP connection your server accepts consumes one FD. The default per-process limit on Linux is 1,024. When that fills, the OS returns EMFILE (too many open files) and refuses new connections — regardless of how much CPU or memory you have left.

$ ulimit -n        # check current limit → probably 1024
$ ulimit -n 65535  # raise it (current session only)
# Permanent: edit /etc/security/limits.conf

TCP Connections: The Handshake

Every HTTP connection starts with a TCP three-way handshake before any data flows:

Client → Server: SYN      ("I'd like to connect")
Server → Client: SYN-ACK  ("Got it, I'm ready")
Client → Server: ACK       ("Confirmed — send your request")
Client → Server: [data]    (the actual HTTP request)

Each step takes a network round-trip. For a client in the same data center that's ~0.1ms; across continents it's 100–300ms. This overhead motivated HTTP Keep-Alive.

Keep-Alive: Efficiency With a Hidden Cost

HTTP Keep-Alive reuses a single TCP connection for multiple requests instead of handshaking for each one:

Without: handshake → request → close,  handshake → request → close  (2 handshakes)
With:    handshake → request → request → request → close             (1 handshake)

Faster for the client. But idle keep-alive connections still hold an FD open on the server doing nothing. 5,000 users sitting on an idle browser tab = 5,000 FDs consumed. This is a trap that ended the startup in the war story below.

TIME_WAIT: The Port Cooldown

When a TCP connection closes, the port on the client side enters TIME_WAIT for 60–120 seconds. During this window, that port can't be reused for new outbound connections.

1,000 outbound connections/sec × 60 sec TIME_WAIT = 60,000 ports locked
Total available ephemeral ports: ~65,000
→ Port exhaustion, even with healthy CPU and memory

This matters most on servers that make many outbound connections (proxies, services that call many APIs). You'll see connect: Cannot assign requested address in logs.

The Resource Landscape

A server has five independent ceilings, each with its own failure signature:

Resource	What fills it	Failure signature	Default limit
File descriptors	TCP connections, open files	`EMFILE` — new connections refused	1,024/process
Memory	Heap, thread stacks, buffers	OOM Killer silently kills process	Depends on RAM
CPU	Computation, context switching	All requests slow down together	100% across cores
Disk IOPS	Read/write operations	Write latency spikes	SSD: ~100K–500K/s
Network ports	Outbound connections (TIME_WAIT)	New outbound connections fail	~65K ephemeral ports

The critical distinction: CPU and memory saturation cause gradual degradation — everything slows down proportionally. FD and port exhaustion cause total failure — new connections are refused while existing ones and dashboard metrics look completely healthy. The second kind is the one that produces a 47-minute outage where nobody can figure out why the server is "down."

Concurrency Models: How the Server Handles Multiple Users

How a server handles multiple simultaneous connections determines which ceiling it hits first. There are three main approaches.

Thread-per-Connection

A new OS thread is created for each incoming connection. That thread handles everything for that connection — reading the request, processing it, writing the response — then terminates.

User A connects → Thread 1 created (1MB RAM)
User B connects → Thread 2 created (1MB RAM)
User C connects → Thread 3 created (1MB RAM)
...
User 32,000 → Thread 32,000 → OOM

Simple to code: each thread runs straight-line blocking code. But each thread costs ~1MB of stack space, and context switching between thousands of threads burns CPU. At 10,000 threads, 10–30% of CPU goes just to saving and restoring thread state. At 100,000 — the system is unusable.

First ceiling: Memory (32GB ÷ 1MB/thread ≈ 32,000 max threads), then CPU from context switching.

Event Loop

One thread handles all connections by processing callbacks as they become ready. Instead of blocking while waiting for I/O, it registers a callback and moves on.

Hotel receptionist analogy:
  Loop forever:
    1. Any new guests?        → start check-in
    2. Any responses ready?   → send them
    3. Any room service done? → notify the room (non-blocking)
    4. Back to step 1

ONE receptionist. MANY guests. Never stands idle waiting.

Each connection costs only a few KB (just a file descriptor and some socket state), not 1MB. The same 32GB server can hold 100,000+ connections.

First ceiling: File descriptors (OS limit), or CPU if any one request does synchronous computation.

Weakness: Head-of-line blocking. One slow synchronous task holds the single thread, freezing every other connection behind it.

Drive-through analogy:
  Car A: "Custom birthday cake" (10 min)
  Car B: "Just a coffee"  ← waits 10 minutes behind Car A
  Car C: "A muffin"       ← waits 10:30

Thread Pool

A fixed number of pre-created threads pull work from a shared queue. Balances the two extremes: bounded memory cost (N threads, not one per connection), and true parallelism across CPU cores.

Hire 20 waiters. All new tables wait in the lobby.
A free waiter → grabs next table → serves → returns to lobby.
If all 20 are busy → table waits in queue.
If queue is full → reject with 503.

The pool size formula:

pool_size = num_cores × (1 + wait_time / compute_time)
Example: 8 cores, 90ms DB wait, 10ms compute → 8 × 10 = 80 threads

Failure mode: If a slow downstream dependency holds all threads (e.g., 200 threads × 30-second timeout = all blocked), the queue fills and the service rejects everything — even though your code is fine.

The Comparison That Matters

	Thread-per-Connection	Event Loop	Thread Pool
Memory @ 10K connections	~10 GB	~50 MB	~500 MB
Max practical connections	1K–10K	10K–100K+	10K–50K
Failure mode	OOM / context-switch death spiral	Head-of-line blocking	Queue overflow
Race conditions	Every CPU instruction boundary	Only at `await` boundaries	Every CPU instruction boundary
Who uses it	Apache httpd	Node.js, nginx, Redis	Java Tomcat, Go/Netty

The "Pure Event Loop" Lie

Node.js is described as single-threaded. It isn't, exactly. The event loop handles network I/O directly (non-blocking), but file I/O and DNS lookups are blocking operations that can't be done asynchronously by the OS. Node offloads these to a small internal thread pool via libuv:

Event loop handles:          libuv thread pool (default: 4 threads) handles:
  TCP connections              fs.readFile / writeFile
  HTTP parsing                 DNS lookups (dns.lookup)
  DB queries over network      crypto (pbkdf2, scrypt)
  setTimeout / setInterval     zlib compression

The 4-thread default is dangerously low in production. If 4 file reads are in progress, a DNS lookup queues behind them. HTTP calls to external APIs start timing out for no obvious reason. The fix: UV_THREADPOOL_SIZE=64 set before the process starts.

Every "event loop" runtime is actually a hybrid. Knowing which operations go to the thread pool and which stay on the event loop is essential for debugging latency spikes.

What the Experiments Showed

The prototype: a Node.js HTTP key-value store, raw http module, zero npm packages. Four routes: /data/:key (read/write), /stats (live metrics — memory, FDs, event loop lag), and /heavy (a deliberate CPU blocker). Simple enough to break in predictable ways.

Experiment 1 — Connection flood: the ceiling was the OS kernel

Opening TCP connections without closing them, stepping up in batches. The server stayed alive until:

15,965 connections → ENOBUFS (no buffer space available)
CPU: ~0%    Node heap: ~480 MB (healthy)

The event loop was not blocked. Node's memory was fine. What ran out was OS kernel network buffer space — roughly 250 MB of non-paged pool memory consumed by socket receive/send buffers (~16 KB each). The ceiling was in the kernel, not in the application.

Ceiling hierarchy hit in order:
  1. OS kernel network buffers  ← actual ceiling on Windows
  2. FD limit (~16K on Windows) ← would have been next
  3. Node.js heap               ← never reached
  4. CPU                        ← never reached

This is the war story that opens the phase: a social media API during a viral spike. CPU at 5%, memory at 40%, every dashboard looked healthy — but the server was refusing every new connection. 47-minute outage. The fix:

ulimit -n 65535   # raises the per-process FD limit from the default 1,024

The point isn't to memorize ulimit. It's that the server had massive headroom on four of its five ceilings, and failed on the fifth one nobody was measuring.

Experiment 2 — One CPU-bound request freezes all 15,965 users

A single request to /heavy:

// This holds the event loop thread — nothing else can run
const start = Date.now();
while (Date.now() - start < 5000) {}

During those 5 seconds:

/stats response time:  5,000 ms  (normally 1 ms)
Event loop lag:        12,000 ms
CPU:                   98%
All other requests:    frozen

The /stats route does trivial work — read process memory, count handles, respond. Under normal conditions it takes 1ms. The while loop on /heavy blocked it for 5 full seconds because the event loop is a single thread with cooperative scheduling. There is no OS scheduler interrupting a running synchronous task to let another request proceed. The CPU-bound route held the thread until it finished.

This surfaces in production from: JSON.parse() on a 50 MB payload, an unanchored regex on user input (ReDoS), or a forgotten fs.readFileSync() in a request handler. All of them stall every connected user simultaneously.

Experiment 3 — Async doesn't make reads safe

100 concurrent PUT requests, each reading the current counter, incrementing it, and writing it back. Expected final value: 100. Actual: 60-something.

The interleaving that causes this:

Request A: await readStore()   → gets { counter: 1 }  ← yields here
Request B: await readStore()   → gets { counter: 1 }  ← stale; A hasn't written
Request A: await writeStore({ counter: 2 })
Request B: await writeStore({ counter: 2 })            ← overwrites; should be 3

The assumption going in: single-threaded event loop means no interleaving. That's true between await calls — code between two awaits is atomic. But the full async function is not. Every await is a yield point where the event loop can run another callback. In a read-modify-write pattern, that's enough.

The same lost-update problem exists in multi-threaded code, just at every CPU instruction instead of only at await boundaries. The event loop makes races less frequent, not impossible.

The Fix Reveals the Lesson

Each experiment has a distinct fix:

FD/buffer exhaustion → raise ulimit -n / set nofile in systemd unit or container config
Event loop blocking → offload synchronous CPU work to worker_threads; never call blocking APIs in a hot path
Async race conditions → serialize writes through a queue, or use atomic DB operations (transactions, compare-and-swap)

All three experiments produced similar surface symptoms — slow or unresponsive server. The root causes are completely different. Applying the wrong intervention (or adding a second server) wouldn't have helped. The resource that fails determines both the symptom and the remedy.

What This Phase Actually Teaches

Not all resource ceilings behave like slowness. FD and port exhaustion produce hard refusals while CPU and memory metrics look healthy. This class of failure is disproportionately misdiagnosed because teams are watching the wrong gauges.

The concurrency model determines which ceiling you hit first. Thread-per-connection hits memory first (~1MB/thread → ~32K threads on 32GB RAM). Event-loop hits file descriptors or CPU-bound blocking first. The model isn't just a performance choice — it changes the failure mode.

What This Phase Actually Teaches

1. A server has five independent resource ceilings, and the smallest one is the actual limit. CPU saturation causes gradual degradation. FD exhaustion and port exhaustion cause total failure while all other metrics look fine. Know which ceiling you're approaching before an incident.

2. The thread-per-connection model kills itself on memory and context switching. At 10,000 threads: ~10 GB of stack space, and 10–30% of CPU wasted on saving/restoring thread state. This is why Node.js, nginx, and Redis all chose the event loop.

3. The event loop trades memory efficiency for CPU vulnerability. A single synchronous task holds the thread. Head-of-line blocking isn't a bug to fix — it's a fundamental property of cooperative concurrency. The architecture choice is: which failure mode can I tolerate? OOM death spiral, or one hot request freezing everyone?

4. "Single-threaded" is a simplification. Node.js has 4 libuv worker threads by default handling file I/O, DNS, and crypto. DNS lookups queue behind file reads. In production, set UV_THREADPOOL_SIZE=64.

5. Async code has race conditions at await boundaries. Code between two awaits is atomic. A full async function is not. Every read-modify-write pattern across awaits needs explicit serialization.

When to use which model:

< 5K connections, simple code needed        → Thread-per-connection
> 10K connections, mostly I/O-bound         → Event loop
Mixed CPU + I/O, need parallelism           → Thread pool
Production at scale                         → Hybrid (event loop + worker pool)

Where I'm Still Fuzzy

ENOBUFS vs EMFILE — I hit ENOBUFS at ~16K connections on Windows, which maps to OS kernel buffer exhaustion, not the FD limit. On Linux with default ulimit -n 1024, EMFILE would appear much earlier. Are these always separable in production?
TIME_WAIT at high connection churn — I understand port exhaustion in theory (1K conns/sec × 60s = 60K locked ports), but I haven't built something that actually hits it. That would require a high-outbound-connection scenario — probably relevant in Phase 9 (service decomposition) when services call each other.
libuv thread pool saturation in practice — the 4-thread default is clearly too low, but what does the performance curve look like as you raise it? Is there a point where more threads hurt?

Four experiments worth running:

GET /stats — see live FD count, heap usage, and event loop lag
Hit /heavy in one tab, then hit /stats from another — watch the lag number
Run 100 concurrent writes to a counter and check if the final value matches the expected count
Open thousands of connections without closing them and watch when the OS complains first

The stats endpoint measures event loop lag by scheduling a setTimeout(0) callback and measuring how long before it actually runs. Under normal load: < 5ms. During a CPU-bound request: in the thousands.

Phase 2 moves the problem to multiple machines — which immediately surfaces a new class of failures that single-machine thinking doesn't predict.

Part of a 12-phase series building distributed systems intuition from first principles. Each phase has a running prototype, failure scenarios, and a gate check before moving on.

System Design the Agentic Way — How I'm Using AI Agents to Actually Learn Distributed Systems

HappyBug — Mon, 13 Jul 2026 06:30:28 +0000

System Design the Agentic Way

I wanted to learn system design. Not "memorize CAP theorem for interviews" learn — actually understand why things break at scale and what to do about it.

The problem: most resources give you theory without pain. You read about load balancers without ever watching one fail. You learn about replication without seeing stale data appear in front of you.

So I built something different.

The Idea

What if I had a pair-programmer that:

Starts every topic with a real production failure (so I feel why it matters)
Makes me predict what will happen before I run anything
Forces me to build minimal prototypes, not just read
Then makes me break them deliberately
And finally tests my judgment with ambiguous 3am scenarios

That's what I built — using GitHub Copilot's custom agents and skills as a structured teaching system.

The Curriculum

12 phases. Each one builds on the previous. No skipping.

Phase	What Goes Wrong Without It
1. Single Machine Ceiling	Your server hits 15K connections and starts refusing
2. Stateless Horizontal Scaling	You deploy and lose 47 payments during the restart
3. Data Storage Trade-offs	Redis crashes and your last 30 seconds of writes vanish
4. Caching & Invalidation	Users see stale prices and buy at the wrong amount
5. Async Processing & Queues	Payment service is down for 2 seconds, 500 orders lost
6. Replication & Consistency	Read replicas serve yesterday's data as if it's current
7. Partitioning & Sharding	One customer has 90% of the data, hot partition melts
8. Coordination & Consensus	Two nodes both think they're the leader, split-brain
9. Service Decomposition	Distributed transaction fails halfway, money vanishes
10. Observability	Something is wrong. You have no idea what.
11. Resilience Patterns	One slow dependency freezes your entire service
12. Capacity Planning	Black Friday hits 10x predictions, everything collapses

The Method

Every phase follows the same loop:

Motivate → Model → Decide → Build → Break → Gate

🔥 War Story    → Feel the pain
🧠 Concepts     → Understand the mechanism
⚖️ Trade-off   → Pick an approach (and own the downsides)
🔨 Build       → Minimal prototype with real code
💥 Break It    → Deliberately kill components
🎯 Gate Check  → Scenario-based judgment test

The "Break It" step is where most of the learning happens. My prediction is almost always wrong the first time. That gap — between what I expected and what actually happened — is the lesson.

The Tooling

The whole system runs inside VS Code using GitHub Copilot's custom agent features:

5 agents — instructor, gate-keeper, adversary, reviewer, orchestrator
8 skills — war stories, concept deep-dives, prototype specs, failure labs, etc.
Knowledge files — real production patterns, failure catalogs, anti-patterns
Learning journal — auto-captured insights, predictions, and mistakes

The agents don't give me answers. They ask me to predict, let me fail, then explain why.

Where I Am

I've completed Phase 1 (single machine limits) and Phase 2 (horizontal scaling with Docker + Nginx + Redis). In the next posts, I'll walk through each phase — what I built, what I broke, and what surprised me.

Try It Yourself

The full system is open source:

👉 system-design-agenticway

What I'd Love From You

⭐ Star the repo if this approach resonates
🐛 Open issues if something's broken or confusing
💡 Suggest improvements to the curriculum
📝 Tell me: how do YOU learn system design? What works? What doesn't?

Next up: Phase 1 — The Single Machine Ceiling. I'll show you what happens when 15,965 TCP connections hit one Node.js process, and what it taught me about operating systems.

Follow for the full series. Each post covers one phase of the journey.

Clone it, open in VS Code with Copilot, and start Phase 1. All you need is Node.js and Docker.

What I'd Love From You

⭐ Star the repo if this approach resonates
🐛 Open issues if something's broken or confusing
💡 Suggest improvements to the curriculum
📝 Tell me: how do YOU learn system design? What works? What doesn't?

Next up: Phase 1 — The Single Machine Ceiling. I'll show you what happens when 15,965 TCP connections hit one Node.js process, and what it taught me about operating systems.

Follow for the full series. Each post covers one phase of the journey.

How are people collaborating here for dev challenges? Need advice :)

HappyBug — Thu, 04 Jun 2026 09:18:12 +0000

DEV Community: HappyBug

System Design the Agentic Way — Phase 3: Data Storage Trade-offs

Phase 3: Data Storage Trade-offs

Table of Contents

The Question

The Building Blocks: What "Saved" Actually Costs

Where a Write Actually Goes: The Four Layers

write() vs fsync and the fsync Tax

Process Crash vs Power Loss

The Durability Window

The Write-Ahead Log

Indexes and Write Amplification

B-tree vs LSM: The Storage-Engine Fork

The Trade-off Matrix

Durability Is Not Availability to Reads

The Fix Reveals the Lesson

What This Phase Actually Teaches

Where I'm Still Fuzzy

Try It Yourself

Next

System Design the Agentic Way — Phase 2: Stateless Horizontal Scaling

Phase 2: Stateless Horizontal Scaling

Table of Contents

The Question

The Building Blocks: What "Add a Server" Actually Requires

State: The Thing That Breaks First

The Shared-Nothing Principle

Sticky Sessions: The Tempting Trap

Load Balancing Algorithms

Layer 4 vs Layer 7

Latency vs Throughput

Concurrent Requests Don't Run Sequentially

What the Experiments Showed

Experiment 1 — Kill the shared state: a hang is worse than a crash

Experiment 2 — Restart Redis: did the data survive?

Experiment 3 — Kill the load balancer: the new single point of failure

The Fix Reveals the Lesson

What This Phase Actually Teaches

Where I'm Still Fuzzy

Try It Yourself

Next

System Design the Agentic Way — Phase 1: The Single Machine Ceiling

Phase 1: The Single Machine Ceiling

Table of Contents

The Question

The Building Blocks: What Actually Handles a Connection

File Descriptors

TCP Connections: The Handshake

Keep-Alive: Efficiency With a Hidden Cost

TIME_WAIT: The Port Cooldown

The Resource Landscape

Concurrency Models: How the Server Handles Multiple Users

Thread-per-Connection

Event Loop

Thread Pool

The Comparison That Matters

The "Pure Event Loop" Lie

What the Experiments Showed

Experiment 1 — Connection flood: the ceiling was the OS kernel

Experiment 2 — One CPU-bound request freezes all 15,965 users

Experiment 3 — Async doesn't make reads safe

The Fix Reveals the Lesson

What This Phase Actually Teaches

What This Phase Actually Teaches

Where I'm Still Fuzzy

Next

System Design the Agentic Way — How I'm Using AI Agents to Actually Learn Distributed Systems

System Design the Agentic Way

The Idea

The Curriculum

The Method

The Tooling

Where I Am

Try It Yourself

What I'd Love From You

What I'd Love From You

How are people collaborating here for dev challenges? Need advice :)