DEV Community

Alex Georghiou
Alex Georghiou

Posted on

Circuit Breakers Need Memory, Not Magic

Circuit Breakers Need Memory, Not Magic

A circuit breaker is a state machine that learns from recent failures. If you instantiate a new breaker on every request, you have removed the memory that makes it useful.

A breaker observes outcomes over a window. Once a threshold is crossed, it stops calling a dependency for a period. After that, a limited number of trial calls determine whether the dependency has recovered.

CLOSED -- failure threshold --> OPEN
  ^                             |
  |                             v
  +------ successful probe -- HALF OPEN
                                  |
                                  +-- failure --> OPEN
Enter fullscreen mode Exit fullscreen mode

The interesting design decisions are scope and classification. Should all endpoints share one breaker? Usually not if they have different health characteristics. Should every 4xx response count as a failure? Usually not. Is an application-level timeout a sign the remote service is unhealthy? Often, but the context matters.

Breakers also need to work with deadlines, retries, and bulkheads. A circuit that opens after a storm of aggressive retries may prevent further damage, but it should not be the first or only line of defense.

Record transitions and rejection counts. A breaker silently returning fallbacks can hide an outage for hours. The healthy design makes degraded operation visible and gives operators evidence to act.

Top comments (0)