DEV Community

HappyBug
HappyBug

Posted on

System Design the Agentic Way — Phase 2: Stateless Horizontal Scaling

#systemdesign #distributedsystems #beginners #learning

Phase 2: Stateless Horizontal Scaling

Phase 1 ended at a wall: a single machine has hard ceilings, and no amount of better code moves them. The obvious answer is "add more servers." Phase 2 is about everything that sentence quietly assumes — and the new failure modes that appear the moment a second machine exists.

Table of Contents

The Question

You have one server and you want three. What has to be true about your application before adding the other two actually helps — and where does the bottleneck go once you do?

Phase 2 is an argument that "add more servers" is not an infrastructure task. It's an application-architecture constraint that decides whether the extra machines do anything at all.

The Building Blocks: What "Add a Server" Actually Requires

A war story sets the stakes. A food delivery startup ran its entire API on one server. A routine Friday 6pm deploy meant an 8-second restart. During those 8 seconds: 47 lost payment confirmations, 312 connection resets, and — because the process had to reload a 2.3 GB data.json — a 45-second full recovery. The bill was $23,000 in refunds.

The root cause isn't the deploy. It's that a single machine can't be restarted without downtime and can't survive its own hardware dying. The fix is to run several identical copies. But copies only work if the servers hold no state — and that word hides most of the difficulty.

State: The Thing That Breaks First

Take the Phase 1 server and imagine running three copies of it. Three distinct kinds of state each break in their own way:

Type of state Example Why 3 copies break it
Persistent data data.json on disk Each server has its own copy → writes diverge, reads disagree
In-memory counters activeConnections Each server sees only its own slice → no global view
Coordination state write queue / locks Only valid inside one process → no cross-server ordering

The pattern: anything a server remembers between requests becomes wrong the moment a second server exists, because the next request might land somewhere else.

The Shared-Nothing Principle

The fix is to externalize all state to a shared service — a database, a cache like Redis, a metrics store like Prometheus — so that no server owns anything. The server becomes a pure function:

request in → (read shared state) → (write shared state) → response out
server remembers NOTHING between requests
Enter fullscreen mode Exit fullscreen mode

This is the shared-nothing architecture. Any request can go to any server, because every server is interchangeable. That interchangeability is the entire point — it's what makes servers disposable, and disposable servers are what let you deploy, restart, and lose hardware without an outage.

Sticky Sessions: The Tempting Trap

If a server does hold per-user state (a login session in memory), you can paper over it by pinning each user to the same server every time — a sticky session.

Stateful  + sticky sessions → user X always routed to Server 2
Stateless + any routing     → user X can be served by any server
Enter fullscreen mode Exit fullscreen mode

The trap: sticky sessions reintroduce the single-machine failure. When Server 2 dies, every user pinned to it loses their session — logged out, cart emptied — even though two other servers are healthy. Sticky sessions are a temporary crutch. The goal is always full statelessness, where a dying server costs nothing.

A related temptation is to jump straight to microservices "to reduce blast radius." Scale the monolith horizontally first: three identical copies, deployed one at a time (a rolling deploy), gives zero-downtime with zero architectural change.

Monolith, 1 server      → single point of failure
Monolith, 3 servers     → scales, survives restarts, still SIMPLE
Microservices, 20 svcs  → scales, but adds network failures + distributed transactions
Enter fullscreen mode Exit fullscreen mode

Decompose only when there's a specific reason to (that's Phase 9), not reflexively.

Load Balancing Algorithms

With three interchangeable servers, something has to decide who gets each request. That's the load balancer, and the decision rule matters:

Algorithm How it works Best for Weakness
Round Robin A, B, C, A, B, C… Equal servers, equal-cost requests Ignores how heavy each request is
Least Connections Route to server with fewest active connections Mixed request durations Blind to raw server capacity
Weighted Round Robin More traffic to beefier servers Heterogeneous hardware Static, doesn't adapt to load
Random Pick any server Surprisingly good at scale Can cluster at low request counts

For a server where /heavy takes 5 s but /data/:key takes 5 ms, Least Connections is the right pick. Round Robin would keep handing new requests to a server already stuck on a slow one:

Round Robin:  Server A busy on /heavy (5s) → still gets 2 more → queue builds
Least Conn:   Server A shows 3 active     → skip it → route to Server B (0 active) ✓
Enter fullscreen mode Exit fullscreen mode

Layer 4 vs Layer 7

"Reverse proxy" and "load balancer" are not separate things — any proxy that hides backends can trivially point at several of them. The meaningful distinction is which network layer it operates on:

Layer 4 (transport):  routes packets by IP + port. Fast, protocol-blind.
                      Example: AWS Network Load Balancer.
Layer 7 (application): understands HTTP — can route by path, header, cookie.
                      Example: Nginx, HAProxy, Envoy.
Enter fullscreen mode Exit fullscreen mode

Layer 7 costs a little more per request but can make smart, content-aware routing decisions. This prototype uses Nginx (Layer 7).

Latency vs Throughput

Externalizing state has a cost, and naming it correctly matters. Two independent measures:

Latency    = how long ONE request takes         (a single car driving A → B)
Throughput = how many requests complete per unit (cars arriving per hour)
Enter fullscreen mode Exit fullscreen mode

Adding servers (or highway lanes) does not make a single request faster. It lets the system handle more requests at once. Concretely, moving from a local file to an external store:

Phase 1 (local file) Phase 2 (external store)
Latency ~0.2 ms ~5 ms (25× slower)
Throughput ~5K req/s ~30K req/s (6× more)
Failure tolerance zero survives a server dying

The trade is deliberate: accept higher per-request latency in exchange for far higher throughput and the elimination of the single point of failure. As long as latency stays under the ~100 ms human-perception threshold, users don't notice — but they very much notice an outage.

Concurrent Requests Don't Run Sequentially

A common misread: "if each request now waits 5 ms on the database, 10,000 requests take 50 seconds." That's wrong, and understanding why is the crux of the phase.

An await yields the event loop. While one request waits on the database, the server starts the next:

t=0.00 ms  Req 1 arrives → await db.read() → YIELDS
t=0.01 ms  Req 2 arrives → await db.read() → YIELDS
t=0.02 ms  Req 3 arrives → await db.read() → YIELDS
t=2.00 ms  Req 1's response comes back → resume → reply
t=2.01 ms  Req 2's response comes back → resume → reply
Enter fullscreen mode Exit fullscreen mode

Each request individually experiences 5 ms. They overlap in time. The system does not fall over from per-request latency — it falls over when the database saturates:

3 servers × 10K req/s × 2 DB ops each = 60,000 DB ops/second
Enter fullscreen mode Exit fullscreen mode

And that is the real lesson of horizontal scaling:

BEFORE:  [Server] ← ceiling was here (Phase 1: ~15K connections)

AFTER:   [Server 1] ─┐
         [Server 2] ─┼─→ [Database] ← ceiling moved HERE
         [Server 3] ─┘
Enter fullscreen mode Exit fullscreen mode

Scaling doesn't eliminate bottlenecks. It moves them downstream.

What the Experiments Showed

The prototype makes this concrete: the Phase 1 server rewritten as an "amnesia server" (all fs calls replaced with Redis get/set), packaged into one Docker image, run as three containers behind an Nginx load balancer, all sharing one Redis.

Client → Nginx (:80) → server1:3000 ─┐
                       server2:3000 ─┼─→ Redis (:6379)  ← shared state
                       server3:3000 ─┘
Enter fullscreen mode Exit fullscreen mode

Every response includes served_by: os.hostname() to prove distribution. Three verifications passed first:

  • Load distribution — 10 parallel /stats requests returned 3 different hostnames (a 4-3-3 split).
  • Shared state — a PUT through one server was immediately readable through the other two. Redis is the single source of truth.
  • Server death — docker stop server1, then read: the remaining two servers served all data with zero loss. The server is genuinely disposable.

Then came breaking it on purpose.

Experiment 1 — Kill the shared state: a hang is worse than a crash

docker stop on Redis, then a request. The expectation was an error response. What actually happened:

curl → server → await redis.get() → (never resolves) → request HANGS
Enter fullscreen mode Exit fullscreen mode

No crash, no error — the Redis client silently retries the connection forever, so await redis.get() never settles. This is the worst failure mode: a crash returns an error instantly, but a hang ties up the connection, and enough hangs exhaust Nginx's connection pool and freeze the entire system. Always set timeouts on external service calls. A fast failure is recoverable; a silent hang cascades.

Experiment 2 — Restart Redis: did the data survive?

docker start on Redis, then read the old keys. The data was still there. Redis writes an RDB snapshot to disk on a graceful shutdown (SIGTERM).

The caveat matters: a docker kill (SIGKILL) or a real crash skips the snapshot, so everything since the last auto-save is lost. Graceful shutdown preserved durability here; an ungraceful one wouldn't have. Real durability guarantees are a Phase 3 topic.

Experiment 3 — Kill the load balancer: the new single point of failure

docker stop on Nginx, then curl localhost: connection refused. The servers and Redis were all healthy — reachable directly via docker exec — but unreachable from outside, because Nginx is the only public entry point.

Client → [Nginx]  → Server 1 → [Redis]
          SPOF       Server 2    SPOF
                     Server 3
                    (redundant)
Enter fullscreen mode Exit fullscreen mode

Only the servers gained redundancy. Scaling moved the single point of failure (SPOF) from the server to two new places: the load balancer in front and the shared state behind.

The Fix Reveals the Lesson

Each break has a distinct remedy, and each remedy points at a later phase:

  • State divergence across servers → externalize every kind of state to a shared service; keep the server a pure function.
  • Hang on dependency failure → wrap every external call in a timeout (and eventually a circuit breaker — Phase 11).
  • Load balancer / shared-state SPOF → run multiple load balancers behind a cloud LB, and replicate the shared state (Phase 6).

The surface symptoms differ, but the shape is identical to Phase 1: the component that fails determines both the symptom and the fix. What changed is that "the component" is now something downstream of the server you added — which is exactly what horizontal scaling does to bottlenecks.

What This Phase Actually Teaches

1. Horizontal scaling is an application property, not an infrastructure one. Adding servers only helps if the servers are stateless. The hard work is externalizing state (data, counters, coordination); provisioning the machines is the easy part.

2. Shared-nothing makes servers disposable, and disposable is the whole goal. When any request can go to any server, you can deploy one at a time, tolerate hardware death, and restart freely — the three things a single machine can never do.

3. Sticky sessions quietly reintroduce single-machine failure. Pinning users to servers is a crutch; a dying server takes its pinned users down with it. Full statelessness is the target.

4. The load-balancing algorithm must match request-cost variance. With uniform requests, Round Robin is fine. With a mix of 5 ms and 5 s requests, Least Connections avoids piling work onto an already-busy server.

5. Scaling moves bottlenecks downstream — it never deletes them. Three servers turn one ceiling (connections) into a new one (database throughput at ~60K ops/s) plus two new SPOFs (the LB and the shared state). Know where the ceiling went.

6. A hang is worse than a crash. An unbounded wait on a dead dependency ties up resources and cascades. Every external call needs a timeout.

Decision heuristic:

Need zero-downtime deploys / HA, low complexity   → monolith × N behind a load balancer
Requests vary wildly in cost                      → Least Connections
Uniform requests, equal servers                   → Round Robin
Need path/header/cookie-aware routing             → Layer 7 (Nginx/Envoy)
Need raw packet throughput, protocol-blind        → Layer 4 (cloud NLB)
Independent scaling / team boundaries force it     → microservices (Phase 9), not before
Enter fullscreen mode Exit fullscreen mode

Where I'm Still Fuzzy

  • Timeout values on external calls. I know an unbounded wait is the failure — but what's the right timeout? Too low and you fail healthy-but-slow requests; too high and you hang. It presumably depends on the dependency's p99, but I don't have a principled way to set it yet.

  • Load balancer redundancy in practice. "Run multiple Nginx behind a cloud LB" is the stock answer, but that just pushes the SPOF up to the cloud LB. Where does the regress actually stop — DNS? Anycast? I haven't built the layer above Nginx.

  • When Least Connections misleads. It's blind to server capacity, so a weak server with few connections still looks attractive. In a heterogeneous fleet, when does that go wrong, and is weighted-least-connections the real answer?

Try It Yourself

The prototype is at github.com/aishwarya-chamanoor/system-design-agenticway in phase-02/.

git clone https://github.com/aishwarya-chamanoor/system-design-agenticway
cd system-design-agenticway/phase-02
docker compose up
Enter fullscreen mode Exit fullscreen mode

Four experiments worth running:

  1. Hit http://localhost/stats ten times in parallel — watch served_by cycle across three hostnames.
  2. PUT a value through one request, then GET it repeatedly — every server returns the same value because Redis owns it.
  3. docker stop <server-container> and keep reading — the surviving servers serve everything; the server is disposable.
  4. docker stop <redis-container> and send one request — watch it hang instead of erroring. That hang is the lesson.

Next

Phase 3 goes inside the shared store this phase leaned on — asking what "the data survived" really guarantees, and what it costs to make that guarantee real.

Part of a 12-phase series building distributed systems intuition from first principles. Each phase has a running prototype, failure scenarios, and a gate check before moving on.

Top comments (1)

Collapse
 
xm_dev_2026 profile image
Xiao Man •

The $23K refund story is the kind of postmortem every team needs to read before their first "just add another server" decision. The 2.3 GB data.json reload time is the silent killer that nobody benchmarks until it's 3 AM on a Friday.

What strikes me about the "stateless horizontal scaling as an application-architecture constraint" framing is how directly it maps to the agent world. The same three state categories (session, data, config) that break when you add a second web server also break when you add a second agent instance. Session affinity problems, shared memory assumptions, config drift — they all show up in multi-agent orchestration exactly as you described.

The sticky sessions trap is especially relevant for agents: if an agent's conversation history is in-process memory instead of externalized state, you can't migrate or restart it without losing context. Same problem as your food delivery startup, just with tokens instead of payment confirmations.

Looking forward to Phase 3.