DEV Community

Cover image for Node.js has plenty of circuit breakers. So why did I build another one?
Pedro Rogério
Pedro Rogério

Posted on

Node.js has plenty of circuit breakers. So why did I build another one?

Every service I've worked on eventually grows the same scar tissue: a retry loop copy-pasted into six files, a circuit breaker bolted onto the payment client after an outage, a timeout wrapper someone wrote at 3 a.m. Each one slightly different. None of them talking to each other. And when things go wrong, nobody can answer the only question that matters during an incident: what is the resilience layer actually doing right now?

Java solved this years ago with resilience4j. .NET has Polly. Node.js... has pieces.

The gap

I evaluated what the ecosystem offers before writing a single line:

opossum is the best-known circuit breaker, mature and well maintained. But it's only a circuit breaker — retry is rudimentary, there's no bulkhead, no composition. Metrics need a plugin.

cockatiel is the closest thing to Polly: retry, breaker, timeout, bulkhead, composition. I genuinely like its design. But observability is where it stops — no native metrics, no pipeline-wide correlation — and maintenance has slowed.

The Sindre micro-libs (p-retry, p-timeout, p-limit) are excellent at exactly one thing each. But resilience is a system: a retry that doesn't know the circuit is open will happily sleep through backoff to hammer a dead dependency. Isolated pieces can't coordinate.

And there was one thing nobody documented properly, which became the reason I finally started typing:

Ordering is the whole game

Take four policies: retry, circuit breaker, timeout, fallback. The same four, nested in two different orders, produce two very different systems:

retry( circuitBreaker( timeout( fn ) ) )   // A
circuitBreaker( retry( timeout( fn ) ) )   // B
Enter fullscreen mode Exit fullscreen mode

In A, every attempt flows through the breaker, so the breaker sees the dependency's true failure rate — and when the circuit opens mid-retry, the retry finds out immediately.

In B, the breaker sees one outcome per retry cycle: three real failures against the dependency count as a single failure. The circuit opens far later than the dependency's actual state justifies, and the retry keeps sleeping through backoff against a broker that's clearly down.

Most libraries let you build either one without ever telling you there's a difference. That's not an API problem — it's a documentation-of-consequences problem. I wanted a library where composition and its consequences are the headline feature, not an afterthought.

So I built breakwater.

What it looks like

npm install breakwater
Enter fullscreen mode Exit fullscreen mode
import { resilience, exponential } from 'breakwater'

const payments = resilience({
  retry: { attempts: 3, backoff: exponential({ initial: 200 }) },
  rateLimit: { limit: 100, interval: 60_000 },
  bulkhead: { concurrency: 20, queue: 50 },
  circuitBreaker: { name: 'payments-api', failureThreshold: 0.5 },
  timeout: 2_000,
  fallback: () => ({ status: 'pending', queued: true })
})

const charge = await payments.execute(({ signal }) => api.post('/charge', body, { signal }))
Enter fullscreen mode Exit fullscreen mode

resilience() is the batteries-included path with a fixed, documented order:

fallback( retry( rateLimit( bulkhead( circuitBreaker( timeout( fn ) ) ) ) ) )
Enter fullscreen mode Exit fullscreen mode

Each position is deliberate. The local guards (rate limit, bulkhead) sit outside the breaker, because your own saturation must never open a circuit that describes the dependency's health. The timeout sits innermost so every attempt gets its own budget — and a hung call becomes a countable failure instead of an invisible one.

Need a different order? compose() reads exactly like the nested calls:

import { compose, retry, circuitBreaker, timeout } from 'breakwater'

const policy = compose(
  retry({ attempts: 3 }),          // outermost
  circuitBreaker({ name: 'api' }),
  timeout(2_000)                   // innermost
)
Enter fullscreen mode Exit fullscreen mode

And because the result of compose() is itself a policy, compositions compose again. The ordering guide walks through the classic configurations with sequence diagrams — it's the page I wish every resilience library had.

Details that only show up in production

A few behaviors took far more design effort than any diagram suggests, because they only matter when things are already going wrong:

Policies coordinate through errors. Every error carries a stable code and a retryable flag. CircuitOpenError declares itself retryable: false — so the default retry gives up instantly instead of backing off against an open circuit. Rate-limit and bulkhead rejections stay retryable, because saturation is transient. This is why the pieces compose correctly out of the box.

Cancellation is not failure. One AbortSignal — combining your external cancellation, timeouts, everything — reaches your function. And when the caller aborts, nothing counts it as a failure: retry doesn't retry it, the breaker doesn't tally it, fallback doesn't replace it. Your user closing a tab should never open a circuit.

Timeouts don't lie. A genuine domain error that lands after the deadline propagates as itself, never masked as a TimeoutError. A call the caller cancelled is cancellation, not a timeout. Your error dashboards reflect what actually happened.

Monitoring never changes an outcome. An event listener or metrics collector that throws is isolated and reported — a bug in your telemetry cannot turn a successful payment into an error.

Observability without plugins

Every policy emits typed events, one correlationId crosses the whole pipeline, and stateful policies expose stats():

breaker.stats()
// { state: 'open', failureRate: 0.85, lastError, openedAt, nextAttemptAt, ... }
Enter fullscreen mode Exit fullscreen mode

CircuitOpenError carries that snapshot too — a proper Retry-After header is three lines. For metrics pipelines, implement one MetricsCollector interface and it wires everywhere; define your policies once in a named registry and every metric comes out labeled:

import { policies } from 'breakwater'

policies.define('payments-api', { retry: { attempts: 3 }, circuitBreaker: {}, timeout: 2_000 })

// anywhere else — same name, same instance, genuinely shared circuit state
await policies.get('payments-api').execute(({ signal }) => api.post('/charge', body, { signal }))
Enter fullscreen mode Exit fullscreen mode

A health endpoint over every policy in your app is a policies.names().map(...) away.

Dogfooding, honestly

I maintain a RabbitMQ client library that had its own hand-rolled circuit breaker and retry loop — the exact scar tissue this post opened with. Migrating it to breakwater deleted the bespoke resilience code, upgraded the publisher to the correct retry-outside-the-breaker ordering, and every one of its existing tests passed unchanged, including destructive reconnection tests against a real broker. That migration shipped; it's running in production today.

It also taught me things no unit test would: that consumers of a hand-rolled retry expect the raw last error (there's a documented unwrap pattern now), and that sync call sites need a recreate-the-policy idiom for breaker resets. Both learnings went straight into the docs.

What it is — and isn't (yet)

breakwater today: retry (backoff strategies + jitter + total deadline), timeout (cooperative and aggressive), circuit breaker (sliding windows, bounded half-open probing, manual isolation), bulkhead, rate limiting (token bucket and exact sliding window), chained fallbacks, composition, typed events, metrics, named policies. Zero runtime dependencies, TypeScript-native, dual ESM/CJS, Node >= 22.

Planned and designed for (the circuit breaker's state store is already pluggable): distributed circuit breaker state over Redis — ten instances of your service agreeing that an endpoint is down, with a single instance elected to probe recovery. Plus ready-made Prometheus/OpenTelemetry collectors and a stale-while-open response cache.

The API is heading to 1.0; until then, minor versions may adjust it.

Try it

If you've ever debugged a retry storm at 3 a.m., I'd genuinely love your feedback — issues and PRs welcome. And if breakwater saves you from writing one more bespoke circuit breaker, it did its job.

Top comments (0)