- Book: System Design Pocket Guide: Fundamentals
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
Your LLM provider has a bad afternoon. Completions that took 2 seconds now take 30. Nothing errors and nothing returns a 500; the calls just sit there. Ten minutes later your on-call channel lights up with "nobody can log in." The login route never calls the LLM, but it is waiting in the same line as every chat request, and that line stopped moving.
The fix has a name older than your stack: the bulkhead. You split your capacity into compartments so a flood in one stays in one. This post builds one in TypeScript, about 60 lines with no dependencies, plus a fake slow upstream and a load script so you can watch login p99 before and after instead of taking my word for it.
Why login waits for a call it never makes
Be precise about the mechanism, because it isn't the event loop. A Node.js handler awaiting a slow socket costs almost no CPU. The process could hold thousands of those. What runs out is whatever finite slot sits in front of every route.
On Cloud Run that slot is the per-instance concurrency limit, 80 by default for services deployed from the console. Elsewhere it is your reverse proxy's connection cap, a load-shedding middleware, a fixed worker count, or a shared database pool that a handler holds while it waits for the model. Slow calls take slots and hold them for 30 seconds instead of 2. New chat requests keep arriving and take the next free slots. Within a minute every slot belongs to a request waiting on the same broken dependency, and a login that needs 2 milliseconds of work stands in the queue behind them.
Little's law puts a number on it: requests in flight equal arrival rate times time in the system (L = λW). Hold the arrival rate fixed, multiply W by 15, and you need 15 times the slots. You don't have them.
The outbound side has the same problem one layer down. Node's http.Agent defaults to maxSockets: Infinity per host, and every client request that doesn't name an agent goes through http.globalAgent. One slow upstream can open as many sockets as it likes, because nothing tells it to stop.
The bulkhead idea: a breach floods one compartment, not the ship.
The bulkhead: a limit, a short queue, and a fast no
Microsoft's Azure Architecture Center describes the pattern as isolating "the elements of an application into pools so that if one element fails, the others continue to function," and one of its examples is a consumer that keeps a separate connection pool per downstream service. Resilience4j ships the same idea for the JVM as a semaphore bulkhead with two knobs: maxConcurrentCalls (default 25) and maxWaitDuration (default 0, so a full bulkhead rejects at once).
Here is that in TypeScript. Three numbers per dependency: how many calls may run, how many may wait, and how long a waiter may wait before it gets rejected.
// bulkhead.ts
export class BulkheadFullError extends Error {
constructor(bulkhead: string) {
super(`bulkhead "${bulkhead}" is full`);
this.name = "BulkheadFullError";
}
}
type Waiter = {
resolve: () => void;
timer?: ReturnType<typeof setTimeout>;
};
type Options = {
name: string;
limit: number; // calls in flight
maxQueue: number; // calls allowed to wait
maxWaitMs: number; // how long one may wait
};
export class Bulkhead {
private active = 0;
private queue: Waiter[] = [];
private opts: Options;
constructor(opts: Options) {
this.opts = opts;
}
async run<T>(fn: () => Promise<T>): Promise<T> {
await this.acquire();
try {
return await fn();
} finally {
this.release();
}
}
private acquire(): Promise<void> {
const { name, limit, maxQueue, maxWaitMs } = this.opts;
if (this.active < limit) {
this.active++;
return Promise.resolve();
}
if (this.queue.length >= maxQueue) {
return Promise.reject(new BulkheadFullError(name));
}
return new Promise((resolve, reject) => {
const w: Waiter = { resolve };
if (Number.isFinite(maxWaitMs)) {
w.timer = setTimeout(() => {
this.queue.splice(this.queue.indexOf(w), 1);
reject(new BulkheadFullError(name));
}, maxWaitMs);
}
this.queue.push(w);
});
}
private release(): void {
const next = this.queue.shift();
if (next) {
clearTimeout(next.timer);
next.resolve(); // hand the slot straight over
} else {
this.active--;
}
}
}
A few details carry the weight.
When a call finishes and someone is waiting, release hands the slot directly to the waiter and leaves active alone. Decrementing and letting the waiter increment would open a gap where a brand-new caller could jump the queue.
A rejected acquire throws before the try, so release never runs for a call that never got a slot. A waiter that times out is removed from the queue first, so release can never hand a slot to a promise that already rejected. Both paths run synchronously on the event loop, so there is no race between the timer and the hand-off.
The Number.isFinite check is there for a reason. Node's setTimeout treats any delay above 2147483647 as 1 millisecond, so setTimeout(fn, Infinity) fires almost at once. Without the guard, an "unbounded" wait would reject every waiter right away.
There are no parameter properties and no enums, so the file runs under Node's built-in type stripping, on by default since v22.18.0 and v23.6.0. No build step.
A fake slow provider and an app with two routes
Make a folder, then npm init -y and npm pkg set type=module. The upstream answers every request after DELAY_MS:
// upstream.ts
import http from "node:http";
const DELAY_MS = Number(process.env.DELAY_MS ?? 30_000);
http
.createServer((_req, res) => {
setTimeout(() => res.end("a slow answer\n"), DELAY_MS);
})
.listen(4001, "127.0.0.1");
The app has /login, which does nothing but answer, and /chat, which calls the upstream. A server bulkhead with an unbounded queue stands in for your platform's per-instance cap of 50 slots. MODE=shared sends chat through the global agent with no limit. MODE=bulkhead gives the upstream its own compartment: 10 in flight, 20 waiting, one second of patience, and its own http.Agent capped at the same 10 sockets.
// app.ts
import http from "node:http";
import { Bulkhead, BulkheadFullError } from "./bulkhead.ts";
const MODE = process.env.MODE ?? "shared";
const UPSTREAM = "http://127.0.0.1:4001/complete";
const slots = new Bulkhead({
name: "server", limit: 50,
maxQueue: Infinity, maxWaitMs: Infinity,
});
const llm = new Bulkhead({
name: "llm", limit: 10, maxQueue: 20, maxWaitMs: 1_000,
});
const llmAgent = new http.Agent({
keepAlive: true, maxSockets: 10,
});
function get(url: string, agent?: http.Agent) {
return new Promise<string>((resolve, reject) => {
const req = http.get(url, { agent }, (res) => {
let body = "";
res.setEncoding("utf8");
res.on("data", (chunk) => (body += chunk));
res.on("end", () => resolve(body));
res.on("error", reject);
});
req.on("error", reject);
});
}
function complete(): Promise<string> {
if (MODE === "shared") return get(UPSTREAM);
return llm.run(() => get(UPSTREAM, llmAgent));
}
async function handle(
req: http.IncomingMessage,
res: http.ServerResponse,
) {
if (req.url === "/login") return void res.end("ok\n");
if (req.url !== "/chat") {
res.writeHead(404);
return void res.end();
}
try {
res.end(await complete());
} catch (err) {
const full = err instanceof BulkheadFullError;
res.writeHead(full ? 503 : 502, { "retry-after": "2" });
res.end(full ? "llm busy\n" : "upstream error\n");
}
}
http
.createServer((req, res) => {
slots.run(() => handle(req, res)).catch(() => res.destroy());
})
.listen(3000, "127.0.0.1");
The 503 carries Retry-After, the standard way to tell a client how long to back off. A fast, honest "busy" beats a 30-second hang that ends in a gateway timeout anyway.
The agent's maxSockets matches the bulkhead's limit on purpose. Requests past maxSockets wait inside the agent's own requests queue, and none of the Agent constructor options caps that queue or times it out. If the bulkhead admits at most 10 calls, the agent never has to queue anything. If you call upstreams with the global fetch, note that it runs on undici and takes an undici-compatible dispatcher, not an http.Agent. The equivalent there is an undici Agent with connections: 10, passed as the dispatcher option.
Shared slots versus a compartment: the slow dependency can only fill the part of the server you gave it.
The load script
80 chat requests go out at once. One second later, 50 logins go out together, and the script reports their p99.
// load.ts
const BASE = "http://127.0.0.1:3000";
async function timed(path: string) {
const t0 = performance.now();
const res = await fetch(BASE + path);
await res.text();
return { ms: performance.now() - t0, status: res.status };
}
function p99(ms: number[]): number {
const s = [...ms].sort((a, b) => a - b);
return Math.round(s[Math.ceil(s.length * 0.99) - 1]);
}
const chats = Array.from({ length: 80 }, () => timed("/chat"));
await new Promise((r) => setTimeout(r, 1_000));
const logins = await Promise.all(
Array.from({ length: 50 }, () => timed("/login")),
);
console.log("login p99 ms:", p99(logins.map((r) => r.ms)));
const done = await Promise.all(chats);
const count = (s: number) =>
done.filter((r) => r.status === s).length;
console.log("chat 200:", count(200), "503:", count(503));
Run it in three terminals:
node upstream.ts
MODE=shared node app.ts # then MODE=bulkhead
node load.ts
I haven't run this myself, so treat the next two paragraphs as arithmetic and check it on your machine.
In shared mode, 50 chats take all 50 slots and the other 30 queue. The logins arrive at the one-second mark and line up behind those 30. Nothing frees up until the first wave finishes at about 30 seconds, so login p99 should sit around 29,000 ms, the remaining length of one upstream call, for a route that does no work at all. All 80 chats eventually return 200, the last 30 after roughly a minute.
In bulkhead mode, 10 chats run, 20 wait, and the rest get a 503 within milliseconds, which hands their server slots back. The 20 waiters give up after one second. After the first few milliseconds, chats never hold more than 30 of the 50 slots, so logins walk in and p99 should be in single-digit milliseconds. The chat line should read 10 successes and 70 rejections. That looks worse until you remember the alternative was 80 users staring at a spinner and nobody able to log in.
Sizing the compartment
Start from Little's law with healthy numbers. If the LLM normally takes 2 seconds and you want to carry 5 requests per second, you need about 10 slots in flight. When latency jumps to 30 seconds, those same 10 slots carry about one request every three seconds and the rest of the traffic gets turned away fast. That is the bulkhead doing its job: the dependency's bad day gets capped at the capacity you budgeted for it.
Keep maxWaitMs well under your client's timeout, or waiters will time out on the client side while still holding a server slot. Keep the queue short: a queue only helps absorb small bursts, and when latency is the problem, a long queue just turns fast rejections back into slow ones.
Then add the pieces a bulkhead doesn't cover. The Azure guidance suggests combining it with retries, circuit breakers and throttling. A deadline on the upstream call (signal: AbortSignal.timeout(...) on http.request) limits how long one call can hold its slot. A circuit breaker stops sending traffic to a dependency that is failing every call. And count BulkheadFullError as a metric of its own, because a sudden rise in rejections is the earliest signal you will get that a provider is struggling.
If this was useful
A bulkhead is one building block. Whether a slow dependency stays a small incident also depends on how you think about latency, load balancing, rate limits and availability across the whole system. That is the ground the System Design Pocket Guide: Fundamentals covers, written for engineers who want the core building blocks in one short book.
Top comments (1)
Bulkheads are criminally underused in Node services. Everyone reaches for timeouts and retries, but a timeout still lets the slow dependency eat your whole connection pool while it waits. Isolating the pool per dependency is the part that actually protects the login route. One addition from running this in production, put a metric on bulkhead rejections. A rising rejection count is your earliest warning that a provider is degrading, usually minutes before latency alarms fire. It turns the safety mechanism into a monitoring signal for free.