Circuit Breaker: Unboxing — your message broker is already the control plane

Luca Giordano — Thu, 01 Oct 2026 07:36:30 +0000

A recent interview put me in front of this exact use case, and I gave an average answer that left a bad taste in my mouth. It kept nagging me, so I built it five times and measured each one.

"Add a circuit breaker" usually means adding a library to every service that calls the flaky dependency. That works when the caller is one process. It breaks down when the caller is a fleet.

The setup

One producer, one durable RabbitMQ queue, and a fleet of consumers competing for its messages. Each message becomes one call to a third party whose load balancer, health checks and hosts are entirely theirs.

The textbook breaker assumes one caller and one dependency. We have neither:

Many callers, one opinion needed. A breaker per consumer means each forms its own view from its own sample.
Their load balancer, not ours. We see one address, so we can only decide when to stop calling them, not which host to avoid.
Whose failure is it, anyway? A request shed by our own egress proxy can look identical to an upstream failure. With an upstream that was only 300 ms slower, every 5xx our callers saw came from our own proxy: 90,753 local errors in two minutes.

A breaker with no control plane floods

Step 1 has no breaker: every message spends its three delivery attempts against a dead upstream, and about 4,000 messages are dead-lettered in 20 seconds.

Step 2 puts a standard breaker (cockatiel) in every consumer. It does its one job: each replica stops after five failures. But nobody shares the verdict:

one 15-second outage opens the fleet's five breakers about 27 times;
all five agree on the state only 56% of the time;
each half-open probe runs on its own clock;
and 1,577–2,246 messages are dead-lettered from just 52–59 real failures.

That last number is the lesson. Most of the loss came from what "open" did, not from the outage. Every call an open breaker turned away still spent one of the message's delivery attempts, so messages the third party never saw ended up in the dead-letter queue.

What's missing is a control plane: something the fleet shares that says whether to call and who probes.

RabbitMQ already is one

Steps 3 and 4 add no new infrastructure. Everything the fleet needs to agree on is a broker feature:

The fleet needs	RabbitMQ gives it
one probe for the whole fleet	a one-token queue (`x-max-length: 1`, `x-overflow: reject-publish`) as the probe permit. With every replica open, peak probes in flight went from 6 to 1
one replica for jobs only one should do (replaying dead letters)	`x-single-active-consumer`: the broker elects one consumer and hands over when it disconnects
turning a message away without spending its budget	from RabbitMQ 4.3, a requeuing `nack` no longer counts toward `x-delivery-limit`, while a `reject` still does
"open" held somewhere other than process memory	step 4: a consumer switched off, and a token waiting in a delay chain built from per-queue TTLs and dead-lettering, with no plugin and no timer in the process

Plus the thing a separate control plane would not give you: the work itself. While the breaker is open, messages wait durably in the queue that already holds them, instead of being tried, refused and dead-lettered by every replica.

	Dead-lettered in an outage
Step 2 · a breaker in every process	1,577–2,246 in 15 s
Step 3 · coordinated by the broker	0 in 40 s
Step 4 · held by the broker	0, where cockatiel lost 2,745

The breaker got simpler, not bigger: by step 4 it keeps no state of its own.

The version matters. On a broker older than 4.3 every requeue counts toward the delivery limit, and a message waiting out an open breaker gets dead-lettered untried.

When the verdict has to leave the fleet

For most consumer fleets, step 3 or 4 is where to stop. Step 5 is for when other systems need to act on the verdict: a producer that stops accepting work, a status page, billing.

There, Envoy enforces per replica (outlier detection, concurrency limits) and a leader-elected aggregator takes a quorum of what the replicas report. It publishes one verdict per API as events with a gapless sequence. The fleet still coordinates through RabbitMQ: a topic exchange carries each verdict to a queue per consumer, and single-active-consumer queues elect the one consumer that probes, redrives or keeps a minimum of work going.

Across ten chaos faults (killing consumers, the leading aggregator, the broker, a hanging upstream, a lease partition mid-outage) it lost 0 messages and dead-lettered 0. It costs about three times the code of step 3 and 19 containers, and it pays only when something outside the fleet consumes the verdict.

Takeaways

A breaker is four decisions: what counts as a failure, where the state lives, what "open" does, and who decides to try again. A library answers all four together; pulling them apart lets you answer each one better.
For a fleet, "open" should mean stop consuming, not reject the message. Your broker already knows how to hold work.
Look at your broker before adding a coordination service. Single-active-consumer, delivery limits, TTLs and dead-lettering already give you a probe permit, leader election and a durable timer.
Measure what "open" costs, not just whether the breaker trips.

The full article, with each step's design and measurements, is at lsfera.github.io/circuit-breaker-unboxing. The code is TypeScript on Effect 4 and amqplib, one branch per step:

lsfera / circuit-breaker-unboxing

Reasoning over circuit breaker

Abstracting over the circuit breaker

📖 Read the full article: Circuit Breaker: Unboxing

Start with 01 · Baseline(no breaker)

"Add a circuit breaker" usually means adding a library to every service that calls the flaky dependency. Written that way, the breaker looks like one thing. It is really four answers that a library happens to give together:

What counts as a failure. A 503 is the third party failing. A 429 is it asking you to slow down. A 422 is it refusing this one request.
Where the state lives. In each process's memory, in something the fleet shares, or in a control plane.
What "open" does. It rejects calls locally, or it stops taking work at all.
Who decides to try again. A timer in every replica, or one probe the whole fleet agrees on.

With the four pulled apart, each can be given a better answer than the…

View on GitHub

DEV Community: Luca Giordano