Imagine a payments service that starts timing out at 14:02. Your app keeps sending requests. Each one waits two seconds and fails. By 14:04 the worker pool is full, and checkout is down because one dependency is sick.
A circuit breaker stops the sending. After a set number of failures it returns an error without making the call. Your app keeps the capacity it would have spent on calls that cannot succeed.
The calls that cannot succeed
A breaker wraps a remote call and watches the result. Past a threshold it stops calling. Azure's architecture guide states the purpose: the pattern "helps prevent an application from repeatedly trying to run an operation that's likely to fail."
Martin Fowler credits Michael Nygard's book Release It! with popularizing the pattern. In Fowler's account, the breaker trips once failures reach a threshold. Then "all further calls to the circuit breaker return with an error, without the protected call being made at all."
You save the wait, not the error. A failing call still holds a connection and a worker until its timeout fires. Azure warns that blocked requests "might hold critical system resources, such as memory, threads, and database connections." Enough of them and unrelated parts of the app fail too.
Example: a checkout service calls payments with a 2-second timeout. Payments hangs. At 50 checkout requests a second, 100 workers sit waiting on the same broken dependency, and the pool is gone in under a minute.
The breaker sits between the caller and the dependency; when it opens, the call stops there and the error comes back at once.
Closed, open, half-open
The breaker is a small state machine.
Closed. Calls pass. The breaker counts failures inside a rolling window, a moving slice of time such as the last 10 seconds.
Open. Calls fail at once. Nothing reaches the dependency, and a timer runs.
Half-open. When the timer ends, a few trial calls pass. A success closes the breaker and resets the count. A failure reopens it and restarts the timer.
Azure explains why the third state earns its place. A recovering service "might be able to support a limited volume of requests until the recovery is complete." A flood of work can knock it over again.
Failures open the breaker, the cooldown leaves it half-open, and the trial call decides the next state.
What trips it
Start with a count of failures in a window. Azure describes a counter that resets at intervals, so the occasional error does not open the circuit. The threshold trips only when enough failures land inside the period.
Fowler suggests a rate instead: open at a 50% failure rate. Different errors can have different limits, such as "a threshold of 10 for timeouts but 3 for connection failures."
Count only errors that say the dependency is unhealthy. A 404 for a missing record, or a rejected input, is a normal answer. Fowler is direct: "Not all errors should trip the circuit."
Scope the breaker to one dependency, often to one operation. Wide scope causes a problem: a single breaker over every call to a service will open for a fault that affects one endpoint.
a breaker with three states
// A breaker for one dependency. The numbers are examples.
function breaker(fn, { threshold = 3, cooldown = 30000 /* ms */ } = {}) {
let failures = 0;
let openedAt = 0;
return async (...args) => {
const open = openedAt > 0 && Date.now() - openedAt < cooldown;
if (open) throw new Error("circuit open");
try {
const result = await fn(...args);
failures = 0;
openedAt = 0;
return result;
} catch (err) {
if (++failures >= threshold) openedAt = Date.now();
throw err;
}
};
}
The next call after the cooldown is the probe. A success resets the breaker. A failure starts the cooldown again.
What the caller sees
An open breaker returns an error at once. The caller needs a plan for that error. Fowler lists the choices: fail the operation, use a workaround, show stale data, or put the work on a queue for later.
Example: a product page calls recommendations and inventory. With the recommendations breaker open, the page renders without the recommendations strip and keeps the buy button. The inventory call still runs. One broken dependency costs a strip, not the sale.
Hystrix, Netflix's breaker library, calls this fallback and graceful degradation. It also rejects a call when its pool of threads is full, instead of queueing work that will wait.
Where the breaker lives
You can put the breaker in the client. Hystrix, Netflix's library, and resilience4j, a Java library, wrap each call in process. The breaker knows which operation it protects, so per-endpoint thresholds are possible.
You can put it in the network. Envoy's outlier detection is passive health checking: it removes hosts that fail or slow down from the load-balancing set. Envoy caps how many hosts it ejects, and each repeat lengthens the time a host stays out.
The two see different things. A proxy sees host health, not what the operation means. A client library sees the operation, but each instance keeps its own count unless you share state.
It does not fix the service
A breaker protects your capacity. It does not repair the dependency, and it does not make a failed call succeed. The half-open probe exists to notice recovery, nothing more.
Traffic returns when the breaker closes. If every instance probes at the same moment, the recovering service gets a wave. Stagger the probes, or let one instance probe first.
Order matters with retries. Retries must respect the breaker. Azure says retry logic must watch what the breaker returns. Retries should stop "if the circuit breaker indicates that a fault isn't transient."
Log every state change with the failure count that caused it. An open breaker is a signal about the dependency. Fowler argues that operations staff should be able to trip or reset a breaker by hand.
Without a breaker, the slowest dependency decides how much of your capacity it gets. With one, you decide how many calls to waste before you stop.
Takeaway
A timeout bounds one call. A breaker bounds the number of calls, and protects your capacity, not the dependency.
Common questions
Is a circuit breaker the same as a timeout?
No. A timeout bounds one call. A breaker stops new calls to a dependency that keeps failing. Use both: the timeout ends the call, and the breaker ends the attempts.
Which errors should open it?
Only errors that mean the dependency is unhealthy: timeouts, connection errors, and 5xx responses. A 404 or a rejected input is a normal result and should not count.
What happens to calls already in flight?
They finish or hit their timeout. The breaker only blocks new calls.
Sources
Originally published at awahab.dev.
Top comments (0)