Day 8 — Circuit Breaker: Stops the Retry Storm

Retries rescue you from a blip. But when a dependency is fully down, those same retries pile on and become a storm. Today the client learns to stop knocking on a door that won't open — and to notice, on its own, when the door opens again.

🙏 Credit where it's due — This series is my attempt to internalize and share what I learned from Mahmoud Youssef's excellent course, Fundamentals of Distributed Systems on Udemy. The course material, structure, and topic flow are his work; the explanations, code examples, and diagrams in these articles are my own rewrite in my own words. If you find this useful, please consider taking the course — it goes much deeper than these articles can.

Recap — where retries left us

Day 6 taught the client to retry with backoff + jitter. Day 7 taught the server to push back with 429 + Retry-After, and the client (Part 2) to obey it — turning a temporary rejection into an eventual success.

That whole story assumes the dependency is still alive, just busy. Day 8 asks the harder question: what if the dependency is genuinely down? Now retrying isn't patience — it's a hammer.

The one idea to take away

💡 Retries help with temporary failure. A circuit breaker protects the system when failure is sustained. When every call is failing, the smartest thing a client can do is stop calling for a while — fail fast, give the dependency room to recover, and probe gently to see when it's back.

1. The Problem — retries become a storm

Take the polite retry client from Day 7 and point it at a dependency (say a payments service) that is completely down. Retry-After never arrives because the dependency isn't answering at all — it just fails. So the client does what it was told: try, wait, try again, up to its budget.

With 12 concurrent operations and 4 attempts each, that is 12 × 4 = 48 calls slammed into something that is already on the floor. Every call fails. Nothing completes. The retries didn't help — they multiplied the load on a broken dependency exactly when it needed breathing room.

🐛 The retry storm. A naive retrying client treats "down" the same way it treats "busy." Against a sustained outage that means maximum wasted load: 48 dependency calls, 0 completions. This is the trap a circuit breaker exists to stop.

2. Mental Model — the three states

A circuit breaker is a tiny state machine that sits in front of the dependency call. It watches failures and decides whether a call is even allowed to go out.

• ● Closed — Normal. Calls flow through. Consecutive failures are counted.
• ▲ Open — Tripped. Calls fail fast — the dependency isn't touched at all — for a cooldown.
• ◆ Half-Open — After cooldown, one probe is allowed through to test the waters.

The loop closes itself: a probe that fails sends the breaker back to Open for another cooldown; a probe that succeeds returns it to Closed and normal traffic resumes.

The cycle, as a timeline

Here is the actual path the breaker walked in our run (one shared breaker protecting the dependency, with the dependency recovering partway through):

Time	State	What happens
~0 ms	CLOSED	First calls go out. The dependency is down → they fail. After 3 consecutive failures…
~5 ms	→ OPEN	Breaker trips. The next wave of attempts fails fast — no dependency calls made.
~250 ms	→ HALF-OPEN	Cooldown elapsed → one probe allowed. Dependency still down (< 350 ms) → probe fails.
~250 ms	→ OPEN	Failed probe re-opens the breaker for another cooldown.
~500 ms	→ HALF-OPEN	Cooldown elapsed again → one probe allowed. Dependency has recovered (≥ 350 ms) → probe succeeds.
~500 ms	→ CLOSED	Breaker closes. Remaining operations sail through and complete.

💡 One breaker per dependency. All 12 operations share a single breaker, because the thing being protected is the shared dependency — not any one operation. When it trips, it trips for everyone; when it heals, it heals for everyone.

3. What does "fail fast" mean?

"Fail fast" sounds strange at first — why would failing ever be a good thing? The trick is in the word fast.

When the dependency is down, a normal call doesn't fail instantly. It waits — for a connection, for a timeout, for a retry delay — and only then gives up. Multiply that wasted waiting across 12 operations × 4 attempts and you get a pile of slow, pointless calls hammering something that is already broken.

Fail fast means the client's own circuit breaker says "no" immediately, before any network call goes out. The attempt is rejected by a tiny in-memory guard in microseconds, instead of waiting on a dependency that is almost certainly still down.

💡 🏠 Everyday picture. Imagine knocking on a shop door and seeing a "CLOSED" sign in the window. You don't stand there knocking for 30 seconds hoping — you read the sign and move on in a split second. The "Open" circuit breaker is that CLOSED sign: it lets the client give up instantly instead of waiting.

In our run, 22 attempts failed fast while the circuit was Open. Those 22 attempts never touched the payments dependency at all — that is exactly the load the breaker shed.

✅ Why this helps the dependency. Every fast-failed attempt is one less call landing on a service that is trying to restart. Failing fast isn't giving up on the user — it's giving the broken dependency the quiet it needs to actually come back.

4. Why Half-Open exists

If the breaker only had Closed and Open, we'd hit a new problem: once it tripped Open, how would it ever know the dependency came back? If it stays Open forever, it has just turned a temporary outage into a permanent one — now we are the reason nothing works.

But the opposite is also dangerous. If, the moment the cooldown ends, the breaker flung all traffic back at the dependency at once, it could knock a still-fragile service straight back down. We need a gentle, careful test — not a stampede.

Half-Open is that careful test. After the cooldown (OpenDurationMs = 200), the breaker lets exactly one probe request through while everyone else keeps fast-failing:

• If the single probe succeeds → the dependency is back → the breaker closes and normal traffic resumes for everyone.
• If the probe fails → still down → straight back to Open for another cooldown, and we try again later.

💡 🚪 Everyday picture. After the shop's posted closing time, you don't send the whole queue crashing through the door. You send one person to gently try the handle. If it opens, everyone follows. If it's still locked, the queue waits a bit longer. That one careful tester is the Half-Open probe.

In our run this happened twice: the first probe (at ~250 ms) landed while the dependency was still down and re-opened the breaker; a later probe (after the 350 ms recovery) succeeded and closed it. That is the self-healing loop working on its own.

5. The Knobs — and why the timing is what it is

The breaker is hand-written with four visible constants:

Constant	Meaning
MaxAttempts = 4	Retry budget per operation (same as the BEFORE scenario, so the two compare fairly).
RetryDelayMs = 250	Wait between attempts.
FailureThreshold = 3	Consecutive failures that trip the breaker Closed → Open.
OpenDurationMs = 200	Cooldown the breaker stays Open before allowing a Half-Open probe.

The simulated dependency is down while elapsed < RecoveryAfterMs = 350, then it recovers.

Why OpenDurationMs and RecoveryAfterMs were tuned (a teaching choice, not a fudge)

One operation's lifetime is bounded by its retry budget:

operation lifetime ≈ (MaxAttempts − 1) × RetryDelayMs
                   = (4 − 1) × 250 ms
                   = ~750 ms

That ~750 ms ceiling is the whole reason the other two numbers are what they are. For this lesson to actually show a full recovery, the breaker must travel Open → Half-Open → (failed probe) → Open → Half-Open → (successful probe) → Closed before the operations run out of attempts and finish.

• If OpenDurationMs were large (e.g. the 750 ms a real system might use), the cooldown alone would consume an operation's entire lifetime. The operations would all finish while the circuit was still Open — you'd never see a Half-Open probe, never see the circuit close, and never see recovery. The lesson would be invisible.
• So OpenDurationMs = 200 (shorter than 750 ms) lets the breaker cycle through Half-Open at least twice inside the budget.
• RecoveryAfterMs = 350 is placed so the first probe lands while the dependency is still down (proving fast-fail and re-open), and a later probe lands after recovery (proving the close). It sits comfortably below the ~750 ms ceiling, leaving room for completed work after the breaker closes.

⚠️ In production these numbers are bigger. A real breaker might stay Open for many seconds and probe slowly. We shrank the clock here purely so a single short, deterministic run demonstrates the full Closed → Open → Half-Open → Closed cycle. The mechanism is identical; only the dial settings are scaled down for teaching.

6. How the Code Works

Let's walk through the AFTER scenario — Session08CircuitBreakerScenario.cs — one piece at a time. No huge code dumps; just the moving parts and how they fit together. Everything below uses the actual names from the file.

① The constants — the dials of the experiment

Constant	Meaning
OperationCount	How many logical operations fire at once. It comes from the session context (ctx.OperationCount) and is 12 in this run.
MaxAttempts = 4	Each operation may try up to 4 times before it gives up.
RetryDelayMs = 250	The wait between two attempts of the same operation.
FailureThreshold = 3	Three consecutive failures while Closed trip the breaker to Open.
OpenDurationMs = 200	How long the breaker stays Open (fast-failing) before it lets one probe through.
RecoveryAfterMs = 350	The simulated dependency is down before this time and healthy after it.

② The fake dependency — no server, no database

The dependency is not a real payments service. It is a tiny local method, CallDependencyAsync(), living inside the test runner. It does only two things: count itself, and answer "am I up yet?" purely from the clock.

async Task<bool> CallDependencyAsync()
{
    Interlocked.Increment(ref dependencyCallsMade);   // count every real call
    await Task.Yield();
    return clock.ElapsedMilliseconds >= RecoveryAfterMs; // down < 350 ms, up after
}

💡 In-process only. While elapsed time is less than RecoveryAfterMs it returns false (down); once elapsed reaches RecoveryAfterMs it returns true (healthy). There is no network call, no real server, and no database anywhere in this lesson — just a stopwatch and a counter.

③ The three breaker states

The breaker is the small CircuitBreaker class, and its mood is a single enum BreakerState { Closed, Open, HalfOpen }:

• ● Closed — Healthy. Every call is allowed through, and consecutive failures are counted.
• ▲ Open — Tripped. Calls are rejected instantly (fast fail) — the dependency is not touched — for OpenDurationMs.
• ◆ HalfOpen — Testing the water. Exactly one probe is let through; everyone else still fast-fails until that probe resolves.

④ The gate method — ask before you call

The key idea of the whole lesson lives in one method, TryAcquire().

✅ Before calling the dependency, the operation asks the breaker first.

TryAcquire() returns an AcquireResult(bool Allowed, bool IsProbe). The operation then reacts to that answer:

var acq = breaker.TryAcquire();
if (!acq.Allowed)
{
    fastFails++;                 // breaker said NO → fast fail, dependency untouched
    // …wait, then try the next attempt
    continue;
}

depCalls++;                      // breaker said YES → the call reaches the dependency
var ok = await CallDependencyAsync();

• If the breaker says yes (Allowed = true) the call actually reaches CallDependencyAsync().
• If the breaker says no (Allowed = false) the attempt becomes a fast fail — fastFails is incremented and the dependency is never touched.
• When Open and the cooldown has elapsed, TryAcquire() promotes exactly one caller to a probe (IsProbe = true) and flips the state to HalfOpen.

⑤ Recording failures

When a call fails, the operation tells the breaker with RecordFailure(acq.IsProbe). Inside, _consecutiveFailures climbs:

• While Closed, reaching FailureThreshold = 3 consecutive failures trips the breaker to Open and records the moment. Three is the line in the sand: a blip or two is tolerated, but a sustained run of failures means "stop hammering."
• If a Half-Open probe fails (wasProbe = true), the breaker goes straight back to Open for another full cooldown — no need to wait for three more failures; one failed probe is proof the dependency is still down.

⑥ Recording success

A successful call calls RecordSuccess(acq.IsProbe), which:

• Resets _consecutiveFailures back to 0 and sets the state to Closed.
• If the success was a Half-Open probe, it also flips ClosedAfterRecovery = true — that is the breaker noticing, on its own, that the dependency came back. A good probe closes the circuit and normal traffic resumes.

⑦ The retry loop — every attempt asks first

Each operation runs the same loop up to MaxAttempts = 4 times. The crucial detail: every attempt begins by asking the breaker, so the same attempt can become either a real dependency call or a fast fail depending on the breaker's current state.

One attempt, step by step:

1. breaker.TryAcquire() — ask permission.
   • Not allowed? fastFails++, wait RetryDelayMs, go to next attempt. Dependency untouched.
   • Allowed? depCalls++, call CallDependencyAsync().
     • Success → RecordSuccess → mark completed → break (stop early, no more attempts).
     • Failure → RecordFailure → wait RetryDelayMs → next attempt.

So early attempts (breaker Closed) tend to be real dependency calls; the wave right after the breaker trips becomes fast fails; later attempts may be probes. That mix is exactly what the report counts.

⑧ Reading the report numbers

Two independent counters drive the verdict: dependencyCallsMade (incremented inside the dependency) and fastFails (incremented when the gate says no). Here is how the headline numbers fall out:

• BEFORE = 48 dependency calls. With no breaker, all 12 operations always burn their full 4 attempts and every attempt hits the dependency: 12 × 4 = 48.
• AFTER = 25 dependency calls + 22 fast fails. With the breaker in front, only 25 attempts actually reached the dependency; the other 22 were rejected instantly while the circuit was Open. Those 22 calls are the load the breaker shed.
• Why total attempts can be 47, not 48. Every attempt is either a dependency call or a fast fail, so total attempts = 25 + 22 = 47. The baseline is 48 (12 × 4). The "missing" attempt is simply an operation that completed early: once a call succeeds, the loop breaks and the operation does not use its remaining attempt(s).
• This is not a bug. A lower-than-baseline total attempt count is the breaker working as designed — finishing the moment the dependency is healthy instead of mechanically spending all four attempts. Fewer calls and fewer wasted attempts is the win.

💡 Quick check: dependencyCallsMade (25) + fastFails (22) = totalAttempts (47) ≤ baseline (48). If you see 48, no operation finished early; if you see less, at least one did.

7. BEFORE — No Circuit Breaker

File: tools/Wassal.SessionTests/Sessions/Session08NoCircuitBreakerScenario.cs

Command:

dotnet run --project tools/Wassal.SessionTests -- session-08-no-circuit-breaker

What it does: fires 12 concurrent operations at a dependency that is always down. Each operation retries naively up to MaxAttempts — no breaker, nothing to stop the calls.

Verified result: 12 operations, each burning all 4 attempts → 48 dependency calls, 0 completed, 12 failed, average attempts/operation = 4.00.

VERDICT: NO CIRCUIT BREAKER RETRY STORM DEMONSTRATED ✅

🐛 Passes by proving the problem. Like every BEFORE diagnostic, success here means the bug was demonstrated: every operation consumed its full retry budget against a down dependency, and all 48 calls were wasted.

8. AFTER — With the Circuit Breaker

File: tools/Wassal.SessionTests/Sessions/Session08CircuitBreakerScenario.cs

Commands (the short session-08 is an alias for the AFTER scenario):

dotnet run --project tools/Wassal.SessionTests -- session-08-circuit-breaker

# Short alias → runs the AFTER (circuit-breaker) scenario:
dotnet run --project tools/Wassal.SessionTests -- session-08

What it does: the same 12 concurrent operations, but every attempt goes through a shared circuit breaker. The dependency is down for the first 350 ms, then recovers. The breaker trips, fast-fails the open-circuit attempts, probes at each cooldown, and closes once the dependency is back.

Verified result:

• 12 operations · 12 completed · 0 failed
• Circuit opened: 2 · Half-open probes: 2 · Closed after recovery: YES · final state: Closed
• Dependency calls made: 25 (vs the 48 baseline) → 23 calls avoided
• Fast-failed attempts: 22 (rejected instantly while the circuit was Open — dependency untouched)

VERDICT: CIRCUIT BREAKER PROTECTED THE DEPENDENCY ✅

✅ The proof is protection + recovery, not perfect completion. The verdict does not require all 12 to finish — under concurrency some operations can exhaust their attempts before recovery. What it requires is that the breaker opened, fast-failed, made fewer calls than the baseline, allowed a half-open probe, closed after recovery, and completed at least one operation. In this run all 12 happened to complete — even better.

BEFORE vs AFTER

Metric	BEFORE — no breaker	AFTER — circuit breaker	Difference
Dependency calls	48	25	23 avoided
Completed operations	0	12	+12
Fast-failed attempts	0	22	load shed
Auto-recovery	none	half-open → closed	self-healing
Verdict	RETRY STORM DEMONSTRATED ✅	DEPENDENCY PROTECTED ✅

Same outage, same retry budget, same 12 operations. The only change is a small state machine in front of the call — and it nearly halved the load on the broken dependency (48 → 25) while taking completions from 0 to 12.

9. What We Proved · What We Didn't Do

✅ What we proved:

• Against a sustained outage, blind retries make 48 wasted calls.
• A circuit breaker trips Open after a few failures and fails fast — sparing the dependency.
• It probes in Half-Open and re-opens if still broken.
• It closes automatically when a probe succeeds after recovery.
• Net effect: 48 → 25 calls (23 avoided) and 0 → 12 completed.

🚧 What we intentionally did not do:

• No Polly — the breaker is hand-written so the states are visible.
• No external package or resilience library.
• No real server dependency — the dependency is simulated in-process.
• No database involved in this lesson.
• No production-ready implementation (no sliding-window stats, half-open concurrency limits, metrics, or per-endpoint breakers).

10. The Important Lesson — three tools, three different jobs

Days 6, 7 and 8 each added a resilience tool. Beginners often blur them together, but they solve different problems and live on different sides of the wire. Here is the one-line difference:

• 🔁 Retry — "try harder" — Lives on the client. Assumes the failure is a brief blip, so it tries again (with backoff + jitter). Great for transient hiccups — but against a real outage it just tries harder at the wrong moment.
• 🚦 Rate limiting — "protect the server from too much traffic" — Lives on the server. The server defends itself from overload by rejecting excess requests (429 + Retry-After). It protects the thing being called from the callers.
• ⛓ Circuit breaker — "protect a failing dependency from being hammered" — Lives on the client. When a dependency is clearly down, the client stops calling it (fail fast) so the broken dependency gets room to recover — then probes gently to detect when it's back.

💡 The mental shortcut: Retry says "keep trying." Rate limiting says "the server protects itself." Circuit breaker says "the caller protects the callee — by knowing when to stop." Retry and circuit breaker are partners: retry handles blips, the breaker steps in when blips turn into an outage.

11. Run It Yourself

① Run the AFTER scenario (the circuit breaker):

dotnet run --project tools/Wassal.SessionTests -- session-08-circuit-breaker

② Compare against the BEFORE scenario (no breaker):

dotnet run --project tools/Wassal.SessionTests -- session-08-no-circuit-breaker

The run prints the per-operation table, the breaker summary, the PASS/FAIL checks, and the final verdict — then exits with code 0 when the proof holds.

③ Where to find the report:

Every run also saves a timestamped text report under the Reports/Session-08/ folder:

Reports/Session-08/Session-08-Circuit-Breaker-RunReport-<timestamp>.txt

The run behind this article is saved as Reports/Session-08/Session-08-Circuit-Breaker-RunReport-2026-06-28-130200.txt.

💡 No setup needed. The dependency, the clock, and the breaker all live inside the test runner. There is no server to start and no database to seed — just run the command and read the report.

Conclusion

A circuit breaker is one of the smallest pieces of code with one of the biggest payoffs in distributed systems. In this lesson, a single hand-written state machine in front of the dependency call turned a wasteful storm — 48 calls, 0 completions — into protected, self-healing traffic: 25 calls, 22 fast fails, 12 completions, and an automatic recovery. The client stopped knocking on a locked door, waited, gently checked the handle, and walked through the moment it opened.

💡 Retries help with temporary failure. Circuit breakers protect the system when failure is sustained. Day 6 made retries gentle. Day 7 made the server push back. Day 8 teaches the client when to stop pushing — and how to notice, by itself, that the dependency has come back.

---

Part of the **Fundamentals of Distributed Systems* series — building Wassal, a distributed food-delivery lab, one concept at a time.*