Every service I've worked on eventually grows the same scar tissue: a retry loop copy-pasted into six files, a circuit breaker bolted onto the payment client after an outage, a timeout wrapper someone wrote at 3 a.m. Each one slightly different. None of them talking to each other. And when things go wrong, nobody can answer the only question that matters during an incident: what is the resilience layer actually doing right now?
Java solved this years ago with resilience4j. .NET has Polly. Node.js... has pieces.
The gap
I evaluated what the ecosystem offers before writing a single line:
opossum is the best-known circuit breaker, mature and well maintained. But it's only a circuit breaker — retry is rudimentary, there's no bulkhead, no composition. Metrics need a plugin.
cockatiel is the closest thing to Polly: retry, breaker, timeout, bulkhead, composition. I genuinely like its design. But observability is where it stops — no native metrics, no pipeline-wide correlation — and maintenance has slowed.
The Sindre micro-libs (p-retry, p-timeout, p-limit) are excellent at exactly one thing each. But resilience is a system: a retry that doesn't know the circuit is open will happily sleep through backoff to hammer a dead dependency. Isolated pieces can't coordinate.
And there was one thing nobody documented properly, which became the reason I finally started typing:
Ordering is the whole game
Take four policies: retry, circuit breaker, timeout, fallback. The same four, nested in two different orders, produce two very different systems:
retry( circuitBreaker( timeout( fn ) ) ) // A
circuitBreaker( retry( timeout( fn ) ) ) // B
In A, every attempt flows through the breaker, so the breaker sees the dependency's true failure rate — and when the circuit opens mid-retry, the retry finds out immediately.
In B, the breaker sees one outcome per retry cycle: three real failures against the dependency count as a single failure. The circuit opens far later than the dependency's actual state justifies, and the retry keeps sleeping through backoff against a broker that's clearly down.
Most libraries let you build either one without ever telling you there's a difference. That's not an API problem — it's a documentation-of-consequences problem. I wanted a library where composition and its consequences are the headline feature, not an afterthought.
So I built breakwater.
What it looks like
npm install breakwater
import { resilience, exponential } from 'breakwater'
const payments = resilience({
retry: { attempts: 3, backoff: exponential({ initial: 200 }) },
rateLimit: { limit: 100, interval: 60_000 },
bulkhead: { concurrency: 20, queue: 50 },
circuitBreaker: { name: 'payments-api', failureThreshold: 0.5 },
timeout: 2_000,
fallback: () => ({ status: 'pending', queued: true })
})
const charge = await payments.execute(({ signal }) => api.post('/charge', body, { signal }))
resilience() is the batteries-included path with a fixed, documented order:
fallback( retry( rateLimit( bulkhead( circuitBreaker( timeout( fn ) ) ) ) ) )
Each position is deliberate. The local guards (rate limit, bulkhead) sit outside the breaker, because your own saturation must never open a circuit that describes the dependency's health. The timeout sits innermost so every attempt gets its own budget — and a hung call becomes a countable failure instead of an invisible one.
Need a different order? compose() reads exactly like the nested calls:
import { compose, retry, circuitBreaker, timeout } from 'breakwater'
const policy = compose(
retry({ attempts: 3 }), // outermost
circuitBreaker({ name: 'api' }),
timeout(2_000) // innermost
)
And because the result of compose() is itself a policy, compositions compose again. The ordering guide walks through the classic configurations with sequence diagrams — it's the page I wish every resilience library had.
Details that only show up in production
A few behaviors took far more design effort than any diagram suggests, because they only matter when things are already going wrong:
Policies coordinate through errors. Every error carries a stable code and a retryable flag. CircuitOpenError declares itself retryable: false — so the default retry gives up instantly instead of backing off against an open circuit. Rate-limit and bulkhead rejections stay retryable, because saturation is transient. This is why the pieces compose correctly out of the box.
Cancellation is not failure. One AbortSignal — combining your external cancellation, timeouts, everything — reaches your function. And when the caller aborts, nothing counts it as a failure: retry doesn't retry it, the breaker doesn't tally it, fallback doesn't replace it. Your user closing a tab should never open a circuit.
Timeouts don't lie. A genuine domain error that lands after the deadline propagates as itself, never masked as a TimeoutError. A call the caller cancelled is cancellation, not a timeout. Your error dashboards reflect what actually happened.
Monitoring never changes an outcome. An event listener or metrics collector that throws is isolated and reported — a bug in your telemetry cannot turn a successful payment into an error.
Observability without plugins
Every policy emits typed events, one correlationId crosses the whole pipeline, and stateful policies expose stats():
breaker.stats()
// { state: 'open', failureRate: 0.85, lastError, openedAt, nextAttemptAt, ... }
CircuitOpenError carries that snapshot too — a proper Retry-After header is three lines. For metrics pipelines, implement one MetricsCollector interface and it wires everywhere; define your policies once in a named registry and every metric comes out labeled:
import { policies } from 'breakwater'
policies.define('payments-api', { retry: { attempts: 3 }, circuitBreaker: {}, timeout: 2_000 })
// anywhere else — same name, same instance, genuinely shared circuit state
await policies.get('payments-api').execute(({ signal }) => api.post('/charge', body, { signal }))
A health endpoint over every policy in your app is a policies.names().map(...) away.
Dogfooding, honestly
I maintain a RabbitMQ client library that had its own hand-rolled circuit breaker and retry loop — the exact scar tissue this post opened with. Migrating it to breakwater deleted the bespoke resilience code, upgraded the publisher to the correct retry-outside-the-breaker ordering, and every one of its existing tests passed unchanged, including destructive reconnection tests against a real broker. That migration shipped; it's running in production today.
It also taught me things no unit test would: that consumers of a hand-rolled retry expect the raw last error (there's a documented unwrap pattern now), and that sync call sites need a recreate-the-policy idiom for breaker resets. Both learnings went straight into the docs.
What it is — and isn't (yet)
breakwater today: retry (backoff strategies + jitter + total deadline), timeout (cooperative and aggressive), circuit breaker (sliding windows, bounded half-open probing, manual isolation), bulkhead, rate limiting (token bucket and exact sliding window), chained fallbacks, composition, typed events, metrics, named policies. Zero runtime dependencies, TypeScript-native, dual ESM/CJS, Node >= 22.
Planned and designed for (the circuit breaker's state store is already pluggable): distributed circuit breaker state over Redis — ten instances of your service agreeing that an endpoint is down, with a single instance elected to probe recovery. Plus ready-made Prometheus/OpenTelemetry collectors and a stale-while-open response cache.
The API is heading to 1.0; until then, minor versions may adjust it.
Try it
- GitHub: github.com/pinceladasdaweb/breakwater
- npm: npmjs.com/package/breakwater
- Start here: the composition & ordering guide
If you've ever debugged a retry storm at 3 a.m., I'd genuinely love your feedback — issues and PRs welcome. And if breakwater saves you from writing one more bespoke circuit breaker, it did its job.
Top comments (0)