Day 9 — Bulkhead Isolation: Contain the Slow Dependency

When every kind of work shares one small pool of slots, a single slow dependency grabs them all — and perfectly healthy work waits behind it. First we build the ship without bulkheads to watch the water spread, then we add watertight walls that contain the leak.

🙏 Credit where it's due — This series is my attempt to internalize and share what I learned from Mahmoud Youssef's excellent course, Fundamentals of Distributed Systems on Udemy. The course material, structure, and topic flow are his work; the explanations, code examples, and diagrams in these articles are my own rewrite in my own words. If you find this useful, please consider taking the course — it goes much deeper than these articles can.

🎯 The one idea to take away: A slow dependency should never be allowed to consume all of your shared resources. If all your work shares one pool of slots, then one slow thing fills the pool — and everything else, healthy or not, has to wait. Bulkhead isolation gives each dependency its own pool, so a slow one can only flood its own compartment.

---

Part 1 — The problem: one shared pool for everything

Real-world analogy — the ship's compartments

A ship's hull is divided into sealed compartments called bulkheads. If one compartment springs a leak, the watertight walls keep the water trapped there — the other compartments stay dry, and the ship stays afloat.

A ship with no bulkheads is the opposite: one leak, and water spreads through the entire hull until the whole ship sinks — even though only one small area was actually damaged.

In software it's the same idea. Without resource isolation, one slow dependency "floods" the shared pool and the delay spreads to unrelated, healthy work. Today we build the ship without bulkheads on purpose — to see the water spread.

One shared pool for everything

Client-side, work usually runs against a limited pool of worker slots (threads, connections, permits). The mistake is letting every kind of dependency call compete for the same pool.

• Different dependency calls share the same limited slots.
• If one dependency turns slow, its calls sit in those slots for a long time.
• Healthy calls — which don't even use the slow dependency — can't get a slot, so they wait.
• The failure spreads from one slow dependency to unrelated healthy work.

💡 The trap: the healthy work isn't slow because it is broken. It's slow because it had to queue behind something else that was broken — all because they shared one pool.

Scenario setup — three kinds of work, one pool

We simulate three kinds of client work, all in-process (no real server, no database). Two are perfectly healthy; one is slow.

Dependency	Role	Operations	Work each
💳 Payment	SLOW dependency	6	700 ms
📦 Inventory	healthy & fast	3	60 ms
🔔 Notification	healthy & fast	3	60 ms

The anti-pattern: one global pool of 4 slots. All three kinds of work share a single SemaphoreSlim(4). Payment starts first and grabs all four slots — so everyone else queues, healthy or not. Payments #5 and #6 queue too, along with all 3 Inventory and all 3 Notification operations, none of which can get a slot for ~700 ms.

💡 Why this is deterministic. Payment is launched first and holds all 4 slots for 700 ms. Healthy work only starts contending ~50 ms later — so it cannot possibly get a slot until a Payment call finishes. Every healthy op is guaranteed to wait.

Timeline — watch the water spread

Here is what actually happens, in order:

• ~0 ms — 6 Payment ops launch. The first 4 grab all shared slots instantly; Payments #5 and #6 queue.
• ~50 ms — 6 healthy ops (Inventory + Notification) arrive — but every slot is already full of Payment. They join the queue.
• ~50–700 ms — The pool stays full of 700 ms Payment work. Healthy work — which only needs 60 ms — sits idle, waiting.
• ~700 ms — The first Payments finish and free slots. Healthy work finally starts… having already waited ~650+ ms for nothing.
• ~725–860 ms — Healthy ops run their tiny 60 ms of work and finish — but their total time was dominated by waiting, not working.

💡 The leak spread. Inventory and Notification were never broken. They were healthy the whole time — they just drowned in the same pool as slow Payment.

The cost — 60 ms of work, ~726 ms of waiting

A healthy operation should take about 60 ms — that's all the work it does. Instead, it spent most of its life stuck in the queue: 726 ms on average, up to 792 ms in the worst case.

⚠️ ~12× slower than it should be. Healthy work that needed 60 ms waited around 726 ms — almost all of that time was pure queuing behind slow Payment calls, not real work.

📊 Report numbers (from the real run):

Metric	Value
Total operations	12
Shared pool size	4 slots
💳 Payment operations	6
📦🔔 Healthy operations (Inventory + Notification)	6
Payment work duration (each)	700 ms
Inventory work duration (each)	60 ms
Notification work duration (each)	60 ms
Average wait — healthy work	726 ms
Max wait — healthy work	792 ms
Healthy ops delayed significantly	6 / 6

All 6 healthy operations completed successfully — they were never broken. They were simply made ~12× slower by sharing a pool with slow Payment work.

Diagnostic verdict: NO BULKHEAD — SLOW DEPENDENCY DELAYED HEALTHY WORK ✅

What this proves:

• The healthy work was not slow because Inventory or Notification were broken — they did their 60 ms of work just fine.
• It was slow because all dependency calls shared the same pool with the slow Payment calls.
• One slow dependency was enough to delay unrelated healthy work by an order of magnitude.
• This is a contagion problem: the failure (slowness) leaked from one place to everywhere through the shared resource.

---

Part 2 — The fix: give each dependency its own bulkhead

What bulkhead isolation means

In software, a bulkhead is the ship-compartment idea applied to resources:

• Each dependency gets its own isolated pool of slots, instead of sharing one global pool.
• A slow dependency can only ever consume its own slots — it can't touch anyone else's.
• The goal is not to make Payment faster. Payment is still slow.
• The goal is to stop Payment from slowing down Inventory and Notification.

💡 Watertight walls between dependencies. If a slow dependency can only flood its own compartment, it can't take the rest of the system down with it. Bulkhead isolation is about limiting the blast radius.

The AFTER setup — three isolated bulkheads

The work is identical to Part 1 — same dependencies, same durations, same counts. The only thing that changes is the pools: instead of one shared pool, each dependency gets its own.

var paymentPool      = new SemaphoreSlim(2);   // Payment's own bulkhead
var inventoryPool    = new SemaphoreSlim(2);   // Inventory's own bulkhead
var notificationPool = new SemaphoreSlim(2);   // Notification's own bulkhead

Payment operations only ever take a slot from paymentPool. Inventory only uses inventoryPool. Notification only uses notificationPool. Nothing crosses between them.

💡 Same head start, opposite result. Just like Part 1, Payment launches first and gets a ~50 ms head start. But this time it fills only its own 2-slot bulkhead — so when the healthy work arrives, the inventory and notification bulkheads are still wide open. The head start no longer matters.

Where the slow work goes now

Before — one shared pool: Payment fills the shared 4-slot pool; everyone else — healthy or not — queues behind it.

After — three isolated bulkheads: each dependency has its own 2-slot compartment. Payment fills its own bulkhead and Payments #3–#6 queue inside the Payment bulkhead only. Meanwhile the Inventory and Notification bulkheads have free slots, so healthy work runs immediately.

💡 The leak is contained. The slow Payment "flood" can only fill the Payment compartment. Inventory and Notification stay dry — their slots were never Payment's to take.

Timeline — the healthy work no longer waits

• ~0 ms — 6 Payment ops launch. The first 2 fill the Payment bulkhead; Payments #3–#6 queue inside the Payment bulkhead.
• ~50 ms — 6 healthy ops arrive. Their bulkheads are empty, so the first 2 Inventory + first 2 Notification ops start immediately.
• ~60–130 ms — Healthy ops finish their 60 ms of work. The 3rd Inventory + 3rd Notification op wait only ~60 ms — for their own bulkhead, not for Payment.
• ~130 ms — All 6 healthy ops are done — while Payment is still grinding through its own queue.
• ~2120 ms — Payment finally finishes its 6 × 700 ms of work (queued behind itself, 2 at a time). Still slow — but it hurt no one else.

💡 The flood stayed in one compartment. Payment took ~2.1 seconds, exactly as slow as before. But the healthy work finished in ~130 ms — it never even noticed Payment was struggling.

The win — 726 ms of waiting → 21 ms

A healthy operation does about 60 ms of work. In Part 1 it spent most of its life queued behind Payment. With bulkheads, the wait nearly vanishes: the average healthy wait drops to 21 ms, and even the worst case (62 ms) is below the operation's own work duration.

✅ ~35× less waiting. Healthy work went from ~726 ms of average wait down to just 21 ms. None of the 6 healthy ops was delayed significantly (threshold: wait ≥ 200 ms).

📊 Report numbers (from the real run):

Metric	Value
Total operations	12
💳 Payment bulkhead size	2 slots
📦 Inventory bulkhead size	2 slots
🔔 Notification bulkhead size	2 slots
💳 Payment operations	6
📦🔔 Healthy operations (Inventory + Notification)	6
Payment work duration (each)	700 ms (still slow)
Healthy work duration (each)	60 ms
Average wait — Payment (its own bulkhead)	705 ms
Average wait — healthy work	21 ms
Max wait — healthy work	62 ms
Healthy ops delayed significantly	0 / 6
Total scenario duration	2120 ms

⚖️ Before vs after — same work, isolated pools:

Metric (healthy work)	BEFORE	AFTER
Avg wait — healthy	~726 ms	21 ms
Max wait — healthy	~792 ms	62 ms
Healthy ops delayed significantly	6 / 6	0 / 6
Payment still slow?	700 ms	700 ms

Payment is exactly as slow in both runs — that's the point. Bulkhead isolation didn't fix Payment; it stopped Payment from hurting the healthy work. The healthy wait collapsed from ~726 ms to 21 ms.

Proof verdict: BULKHEAD ISOLATION — SLOW PAYMENT WAS CONTAINED ✅

What this proves:

• The slow Payment work still ran slowly (700 ms each, ~2.1 s total) — the bulkhead does not fix the slow dependency.
• But Payment was contained to its own pool — it queued only behind itself, never in front of healthy work.
• The healthy work waited an average of just 21 ms (max 62 ms), down from ~726 ms — 0 of 6 healthy ops were delayed significantly.
• One slow dependency could no longer leak across a shared resource and drag down unrelated work.

💡 The lesson behind the lesson: isolation is about limiting blast radius. You may not be able to make a dependency fast — but you can make sure its slowness stays its own problem.

---

Run it yourself

Run the BEFORE / diagnostic scenario:

dotnet run --project tools/Wassal.SessionTests -- session-09-no-bulkhead

Then run the AFTER / proof scenario:

dotnet run --project tools/Wassal.SessionTests -- session-09-bulkhead

Each prints the per-operation table, the summary, the checks, and the verdict — then exits with code 0 when the scenario is proven. Timestamped reports are saved under:

Reports/Session-09/Session-09-No-Bulkhead-RunReport-<timestamp>.txt
Reports/Session-09/Session-09-Bulkhead-RunReport-<timestamp>.txt

💡 No setup needed. The three dependencies and their pools all live inside the test runner. There is no server to start and no database to seed — just run the command.

Bulkhead isolation does not fix the slow dependency itself — Payment was just as slow as before. What it does is limit the blast radius: the slow work is trapped in its own compartment, and the healthy work sails through untouched. One leak should not sink the whole ship.

---

Part of the **Fundamentals of Distributed Systems* series — building Wassal, a distributed food-delivery lab, one concept at a time.*