Day 10 — Timeout & Cancellation: Do Not Wait Forever

Day 9's bulkhead limits how many calls run at once — but never how long a call may hold a slot. When a dependency hangs, its slots stay occupied far too long and everyone queues behind them. First we prove the gap, then we close it with a per-call timeout and cooperative cancellation.

🙏 Credit where it's due — This series is my attempt to internalize and share what I learned from Mahmoud Youssef's excellent course, Fundamentals of Distributed Systems on Udemy. The course material, structure, and topic flow are his work; the explanations, code examples, and diagrams in these articles are my own rewrite in my own words. If you find this useful, please consider taking the course — it goes much deeper than these articles can.

🎯 The one idea to take away: Bulkhead answers "how many calls can run at once?" — Timeout answers "how long is a call allowed to hold a slot?" A bulkhead with no timeout still lets one hung call sit in a slot for as long as the dependency hangs. Limiting concurrency is only half the job.

---

Part 1 — The problem: a hung dependency holds its slots forever

Two different questions

Protecting a dependency call has two independent dimensions. Day 9 answered the first. Day 10 is about the second — the one a bulkhead leaves wide open.

Dimension	Status	What it caps
✅ Bulkhead — how many?	Day 9 · solved	"At most 2 Payment calls run at once." Concurrency is capped by SemaphoreSlim(2).
❌ Timeout — how long?	Day 10 · still missing	"A call may hold a slot for at most 300 ms." Nothing here yet — a hung call holds its slot for 2000 ms.

💡 The gap. A bulkhead caps concurrency, but a hung call still occupies its slot for the dependency's full hang time. Two slots held for 2000 ms each means the next calls simply wait — the bulkhead can't help, because the slow calls are inside the slots, not outside them.

Scenario setup — one hung dependency, its own bulkhead

Everything is simulated in-process (no real server, no database, no external packages). We deliberately keep Day 10 focused on one dependency — Payment — and its own bulkhead. No Inventory, no Notification this time; the whole lesson is about slot-hold time.

• Dependency 💳 Payment is hung — every call takes 2000 ms (simulated with a bounded await Task.Delay(2000, token)).
• It runs through its own bulkhead: paymentPool = new SemaphoreSlim(2) — carried straight over from Day 9.
• We fire 6 Payment operations. There is no per-call timeout.

The Payment bulkhead has 2 slots, and both get stuck on a 2000 ms hang. Payment #1 and #2 hold the slots; Payments #3, #4, #5, and #6 wait in line for a slot to free — which won't happen for ~2000 ms.

💡 Why this is deterministic. 6 calls through 2 slots = 3 waves. Each wave holds both slots for the full 2000 ms hang, so wave 2 starts at ~2000 ms and wave 3 at ~4000 ms. Every number below falls out of that simple arithmetic.

Timeline — three slow waves

• ~0 ms — Wave 1 — Payment #1 and #2 grab both slots and start their 2000 ms hang. #3–#6 queue.
• ~2000 ms — Wave 2 — #1 and #2 finally finish and release. #3 and #4 take the slots and hang for another 2000 ms.
• ~4000 ms — Wave 3 — #5 and #6 finally get in and hang for 2000 ms more.
• ~6000 ms — Everything is done. Total run ≈ 6034 ms for work that should have taken a fraction of that.

💡 The slots never free early. Because there's no timeout, a slot is only released when the 2000 ms hang finally returns. Nothing can shorten it. 0 of 6 slots were released early.

The star metric — slot-hold time

The number that matters in Day 10 is slot-hold time: how long a call keeps its bulkhead slot from the moment it acquires it to the moment it releases it. With no timeout, that equals the full hang — about 2007 ms on average. The calls behind it wait ~2013 ms on average, and up to 4020 ms in the last wave.

⚠️ The slot is the bottleneck. A hung call holds its slot ~2007 ms, so the calls behind it wait ~2013 ms on average and up to 4020 ms. The bulkhead limited concurrency to 2 — it did nothing about the 2000 ms hold.

Per-operation results (from the real run):

#	Dependency	Timeout	Work	Wait	Hold	Total	Reason
1	Payment	none	2000	0	2021	2021	completed
2	Payment	none	2000	0	2018	2018	completed
3	Payment	none	2000	2020	2001	4021	completed
4	Payment	none	2000	2020	2000	4020	completed
5	Payment	none	2000	4020	2000	6021	completed
6	Payment	none	2000	4020	2000	6021	completed

Every call "completed" — but only after holding its slot for ~2000 ms. Waits climb 0 → 2020 → 4020 ms across the three waves.

📊 Report numbers (from the real run):

Metric	Value
Total operations	6
Payment bulkhead size	2 slots
Hung dependency duration (each)	2000 ms
Timeout	none
Average wait (queue for a slot)	2013 ms
Max wait	4020 ms
★ Average slot-hold time	2007 ms
Max slot-hold time	2021 ms
Timed-out operations	0 / 6
Slots released early	0 / 6
Total scenario duration	6034 ms

Diagnostic verdict: NO TIMEOUT — HUNG DEPENDENCY HELD ITS SLOTS AND STALLED EVERYTHING ✅

What this proves:

• The bulkhead did its Day 9 job — never more than 2 Payment calls ran at once.
• But with no timeout, each hung call held its slot for the full 2000 ms — slots never freed early.
• So the 6 calls serialized into 3 slow waves, later calls waited up to 4020 ms, and the whole run took 6034 ms.
• Limiting how many calls run is not enough. You also have to limit how long each one may hold a slot.

---

Part 2 — The fix: timeout + cooperative cancellation

Both dimensions, together

Day 9 and Day 10 are two halves of the same protection. Once you have both, a dependency can neither run away with all your concurrency nor sit forever in the slots it holds.

Dimension	When	What it caps
✅ Bulkhead — how many?	Day 9	At most 2 Payment calls run at once — SemaphoreSlim(2).
✅ Timeout — how long?	Day 10 · this step	A call may hold a slot for at most ~300 ms, then it is cancelled and the slot is released.

✅ Two knobs, two questions. Concurrency is capped by the bulkhead; hold time is capped by the timeout. Neither replaces the other — you want both.

The fix — timeout + cooperative cancellation

The dependency is unchanged: paymentPool = new SemaphoreSlim(2), still 6 calls, each still hanging for 2000 ms. The only new thing is a 300 ms per-call timeout built from a linked CancellationTokenSource.

1. Link a token from the outer (Ctrl+C) token so external cancellation still works.
2. CancelAfter(300 ms) — the linked source will fire the token when the call overruns.
3. Pass the linked token into the dependency call (Task.Delay(HungWorkMs, timeoutCts.Token)), so cancellation is cooperative — the call actually stops when asked.
4. Catch the cancellation and tell our timeout apart from an external cancel, so we report timed-out vs aborted correctly.
5. Release the slot in finally — on every path, success or timeout. This is what frees the bulkhead early.

using var timeoutCts = CancellationTokenSource.CreateLinkedTokenSource(ct);
timeoutCts.CancelAfter(TimeoutMs);

try
{
    await Task.Delay(HungWorkMs, timeoutCts.Token);
}
catch (OperationCanceledException) when (timeoutCts.IsCancellationRequested && !ct.IsCancellationRequested)
{
    // our timeout fired
    result = "timed-out";
}
finally
{
    paymentPool.Release();
}

⚠️ Cancellation must be cooperative. Passing the token into Task.Delay is what makes the hung call actually stop at 300 ms. A timeout you don't pass down is just a stopwatch that nobody obeys — the slot would still be held for 2000 ms.

💡 Always release in finally. If the slot were only released on the success path, a timed-out call would keep its slot for the full hang — silently recreating the Part 1 bug.

Timeline — waves that move fast

• ~0 ms — Wave 1 — #1 and #2 take both slots. At ~300 ms their timeout fires, they abandon and release.
• ~300 ms — Wave 2 — #3 and #4 immediately take the freed slots, time out at ~300 ms more, release.
• ~600 ms — Wave 3 — #5 and #6 get in, time out at ~300 ms, release.
• ~920 ms — All 6 done — each returned a deterministic timed-out result. Total run ≈ 920 ms vs 6034 ms before.

✅ The dependency is still hung. Every call would still take 2000 ms if we let it. We simply stop waiting at 300 ms — so the slots free ~6.5× faster and the whole run finishes in under a second.

The star metric — slot-hold time: 2007 ms → 306 ms

Same metric as Part 1, now bounded by the timeout. A slot is held only until the 300 ms timeout fires, not until the 2000 ms hang returns. Slot-hold drops from 2007 ms to 306 ms, the average wait falls to 309 ms, and the whole scenario finishes in 920 ms.

✅ ~6.5× less slot-hold. The hang is unchanged at 2000 ms, but the slot is held for only ~306 ms — so waits collapse to ~309 ms and the whole run drops from 6034 ms to 920 ms.

Per-operation results (from the real run):

#	Dependency	Timeout	Work	Wait	Hold	Total	Reason
1	Payment	300	2000	0	311	311	timed-out
2	Payment	300	2000	0	310	310	timed-out
3	Payment	300	2000	310	308	618	timed-out
4	Payment	300	2000	310	307	618	timed-out
5	Payment	300	2000	618	299	918	timed-out
6	Payment	300	2000	618	299	918	timed-out

Every call returned a deterministic timed-out result after ~300 ms — not a hang. Waits climb only 0 → 310 → 618 ms, because slots free quickly.

📊 Report numbers (from the real run):

Metric	Value
Total operations	6
Payment bulkhead size	2 slots
Hung dependency duration (each)	2000 ms (still hung)
Timeout	300 ms
Average wait (queue for a slot)	309 ms
Max wait	618 ms
★ Average slot-hold time	306 ms
Max slot-hold time	311 ms
Timed-out operations	6 / 6
Slots released	6 / 6
Total scenario duration	920 ms

⚖️ Before vs after — same hang, bounded hold:

Metric	BEFORE	AFTER
★ Avg slot-hold time	2007 ms	306 ms
Avg wait (queue for a slot)	2013 ms	309 ms
Max wait	4020 ms	618 ms
Timed-out operations	0 / 6	6 / 6
Scenario duration	6034 ms	920 ms
Hung dependency (each)	2000 ms	2000 ms

The hang is identical in both runs — 2000 ms. Timeout did not fix the dependency; it bounded how long a call may hold a slot, so slot-hold dropped from 2007 ms to 306 ms and the run went from 6034 ms to 920 ms.

Proof verdict: TIMEOUT & CANCELLATION — SLOTS RELEASED, SYSTEM STAYED RESPONSIVE ✅

What this proves:

• The dependency is still hung — every call would still take 2000 ms if allowed. Timeout did not heal it.
• But each call now holds its slot for only ~306 ms before the timeout cancels it and finally releases the slot.
• All 6 / 6 calls timed out deterministically and every slot was released — no call hung, no slot leaked.
• The run dropped from 6034 ms → 920 ms. Callers fail fast instead of waiting forever, and the bulkhead recovers instead of strangling itself.

💡 The lesson behind the lesson: a timeout is a promise you make to yourself about how long you'll wait — not a promise the dependency will be quick. Pair it with the Day 9 bulkhead and you've capped both how many calls run and how long each may hold a slot.

---

Run it yourself

Run the BEFORE / diagnostic scenario:

dotnet run --project tools/Wassal.SessionTests -- session-10-no-timeout

Then run the AFTER / proof scenario:

dotnet run --project tools/Wassal.SessionTests -- session-10-timeout

Each prints the per-operation table, the summary, the checks, and the verdict — then exits with code 0 when the scenario is proven. Timestamped reports are saved under:

Reports/Session-10/Session-10-No-Timeout-RunReport-<timestamp>.txt
Reports/Session-10/Session-10-Timeout-RunReport-<timestamp>.txt

💡 No setup needed. The hung dependency, its bulkhead, and the timeout all live inside the test runner. The 2000 ms hang is a bounded delay (never a real infinite wait), and a scenario-level safety cap guards against anything hanging the runner.

The dependency stayed hung at 2000 ms, but our caller stopped waiting at 300 ms, cancelled cooperatively, and released the slot. Bulkhead limits how many; timeout limits how long. Do not wait forever.

---

Part of the **Fundamentals of Distributed Systems* series — building Wassal, a distributed food-delivery lab, one concept at a time.*