DEV Community: Pedro Rogério

My idempotency library had one job. A dropped connection made it run the payment twice.

Pedro Rogério — Mon, 17 Aug 2026 12:32:17 +0000

quayside is my idempotency library for Node.js. The whole API fits in one sentence: execute(key, fn) runs a function exactly once per key — if it already ran, you get the stored result back; if it is running right now, you don't run it again. Pluggable storage (memory, Redis, Postgres, MySQL), HTTP adapters for Express, Fastify, Hono and NestJS, fencing tokens enforced inside the store so a holder that lost its lock can never overwrite a newer execution. Zero runtime dependencies.

It shipped at 1.0.0 last week. This post is about the bug that outlived everything I threw at the code before release — a 100% mutation score across 1,254 mutants, a storage contract suite running against real servers, an adversarial review whose 26 verified findings I fixed or triaged — and waited for the final security pass to be found.

It lived in four lines I had written days earlier, to fix a different bug. And it broke the library on the one scenario idempotency libraries exist for.

the one job

Here is the canonical story every idempotency library tells in its README. A client POSTs a payment. The connection drops before the response arrives. The client has no idea whether the charge happened, so it retries with the same Idempotency-Key. Without protection, the customer gets charged twice. With protection, the retry either waits, gets told the first attempt is still running, or replays the stored response.

quayside's mechanism for this is deliberately boring: the atomic create-if-absent write is the lock. execute(key, fn) writes an IN_PROGRESS record before your function runs — there is no separate locking step, so there is no gap between "checked" and "locked". Success transitions the record to COMPLETED and replays it for a result TTL. Failure deletes it so retries run fresh. A crashed process is handled by the lock TTL. And every transition out of IN_PROGRESS is guarded by a fencing token validated inside the storage — Lua on Redis, token-conditional UPDATEs on SQL — so a holder that stalled through a GC pause and lost its lease gets a FencingError from the store itself instead of silently overwriting the new holder's result.

import { Idempotency } from 'quayside'
import { RedisStorage } from 'quayside/redis'

const idempotency = new Idempotency({
  storage: new RedisStorage(redis),
  resultTtl: '24h',   // how long a completed result stays replayable
  lockTtl: '30s'      // how long a crashed execution blocks the key
})

const result = await idempotency.execute('invoice:123', async () => {
  return createPayment()   // runs once; later calls replay the result
})

Dropped connection, then a retry. That is the use case. Hold that thought.

by the numbers, I had permission to feel safe

I have written before about what mutation testing did to a test suite with 100% coverage, so quayside got the same treatment from the start: every mutant dead, none suppressed — when a mutant was equivalent, the rule was to delete or restructure the code until it wasn't, never to annotate it away.

The storage adapters all pass one shared contract suite against real servers via Testcontainers: 50-way concurrency races, SIGKILL crash recovery, split-brain fencing where a stale holder tries to overwrite a newer execution and must be rejected by the store itself. On top of that, an adversarial review pass: 38 candidate findings, 26 surviving verification, the ten worst fixed before the tag.

I am listing all of this not to brag but to set up the fall. The numbers measure the code you wrote. They say nothing about the code you wrote last Tuesday to fix a review finding, which is where this story actually starts.

the review that found a stuck lock

One of those 26 findings was in the Fastify adapter. Fastify's pipeline is hooks, not a wrapping function, so the adapter bridges the engine's continuation across two hooks: preHandler acquires the lock and stashes a deferred; onSend resolves it with the captured response, which the engine then stores.

The finding: onSend is not guaranteed to run. reply.hijack() — the documented escape hatch for SSE, streaming, proxying — skips it. A handler that never answers skips it. In those cases the deferred never settles, and the key stays locked until the lock TTL expires. Thirty seconds of 409s for every retry, with nothing actually running. A real bug, confirmed, worth fixing.

The fix looked obvious. Node's raw ServerResponse emits close for every request, no matter what the framework's lifecycle does. So: settle the capture when the connection closes.

// onSend never runs for a hijacked reply, an aborted connection or a
// handler that never answers. Without this backstop the capture would
// never settle and the key would stay locked until its TTL expired.
reply.raw.on('close', () => { resolveCapture(null) })

Tests green. Mutation score still perfect. I even wrote a test asserting the new behavior — "a reply that never reaches onSend releases the key on close" — and it passed, proudly.

That test was asserting the bug.

the fix was the bug

The security pass flagged those four lines as its single HIGH finding, and the exploit chain reads like a checklist of things I knew individually and failed to compose:

A client POSTs /payments with Idempotency-Key: K. The handler starts the charge.
Fifty milliseconds in, the client aborts the connection. Maybe a flaky mobile network. Maybe on purpose.
Node does not cancel the handler — the charge keeps running in the background.
But close fires on the raw response immediately. My backstop resolves the capture with null.
To the engine, null means "the response was served but is not cacheable" — the code path built for oversized and binary bodies. It releases the record. Nothing is stored under K. The key is free.
The handler finishes the charge. onSend finally runs, tries to resolve the already-settled deferred — a no-op. No error, no warning. Perfect silence.
The client retries with the same key K. The slot is empty. The handler runs again.

One idempotency key. Two charges. Zero log lines.

And notice what the trigger is: a dropped connection followed by a retry. Not an exotic corner. Not an adversarial contortion. The exact scenario from the README, the one the library has as its single job. The deadlock I fixed was real but bounded — thirty seconds of 409s, annoying, visible. The fix I replaced it with was unbounded and invisible: it spent the library's entire reason to exist, silently, on its most common path.

To be precise about what was at stake: this was caught before it ever reached a registry, in the last review gate before release. Nobody was double-charged. But it survived a mutation score of 100% and a full adversarial code review, because both of those examine the code that exists — and this was a composition failure between a lifecycle event and the semantics of null, each of which was individually correct.

a close is a fact about the connection, not about the work

The actual fix is one condition:

reply.raw.on('close', () => {
  if (reply.raw.writableEnded) resolveCapture(null)
})

writableEnded distinguishes the two worlds that close had been conflating. A hijacked reply finished its response before the connection closed — settle it, release the key, that is the stuck-lock fix still working. A connection that died before the response finished tells you nothing about the handler, because the handler is still running. Those keep the lock.

What happens next is the part I find genuinely satisfying. The still-running handler eventually completes, and onSend — which does run in this case, late — stores the outcome. So the retry that arrives during the charge gets a 409 with Retry-After, and the retry that arrives after it gets the stored response replayed. The late completion is not lost work. It is exactly the result the retry came back for.

If the process dies instead, the lock TTL reclaims the key — the same path that has always governed a crashed execution, tested by the same SIGKILL test. The abort case did not need new machinery. It needed to stop being special-cased into the wrong bucket.

the asymmetry that decides it

The lesson I wrote into the design doc afterwards, in bold, because I intend to re-read it:

Releasing a lock early is an integrity failure. Holding one too long is an availability cost. When the two trade off, hold.

The costs are not symmetric, and the pressure on you is inverted from the risk. A held lock is loud — users see 409s, dashboards light up, someone files an issue, and the damage is bounded by a TTL you chose. An early release is silent — everything returns 200, the work just quietly happens twice, and the bound is whatever a duplicated side effect costs your business. When I wrote the close backstop I was optimizing away the loud, bounded problem, and I paid for it with the silent, unbounded one.

The same asymmetry shows up anywhere a lease, a lock or a saga step can be torn down by an event that merely correlates with the work being done: process managers reaping "idle" workers mid-transaction, orchestrators treating a lost heartbeat as a finished task, HTTP servers treating a gone client as a gone request. A connection event is a fact about the connection. The work has its own lifecycle, and only the work gets to say when it is over.

what else the adversarial passes caught

Three more, briefly, because they rhyme with the theme — every one of them was invisible to coverage and visible to an adversary:

The wait loop trusted the key's identity across time. With onConflict: 'wait', a waiter polls until the holder finishes. But the holder's lock can expire mid-wait and a different payload can take the key over. The waiter was replaying whatever outcome showed up, including one that belonged to someone else's request body. It now re-checks the payload fingerprint on every poll and rejects with the same key-reuse error it would have thrown at the front door.
The HTTP kernel stored control flow as state. "This response must not be cached" used to travel as a sentinel error persisted into the store and deleted right after — and a concurrent waiter could observe it in that window and be waved through with no protection at all. The engine now has a first-class ctx.doNotStore(): keep the lock while running, store nothing, leave nothing for anyone to race. Storing control flow as shared state is a race you just haven't met yet.
A client-supplied key over the length cap returned a 500. It is client input arriving in a header; it is a 400, with a stable error code, like every other policy limit. Blaming the server pages the wrong person.

try it

quayside is at 1.1.0 — the release after 1.0.0 routed persisted failures through the configurable codec too, so if you encrypt results at rest, your error messages and stack traces are no longer sitting next to the ciphertext in plaintext. Everything in this post ships tested: the abort scenario, the takeover scenario, the sentinel window, all pinned by regression tests that fail on the vulnerable versions.

npm install quayside

import { Idempotency } from 'quayside'
import { PostgresStorage } from 'quayside/postgres'

const idempotency = new Idempotency({ storage: new PostgresStorage(pool) })

// core, no HTTP required — queue consumers, cron jobs, CLIs
await idempotency.execute({ payload: message.value, ignoreFields: ['meta.timestamp'] },
  () => processOrder(message.value))

Four storage backends behind one contract suite, HTTP adapters that replay status, headers and body faithfully per the IETF Idempotency-Key draft, Prometheus and OpenTelemetry integrations, zero runtime dependencies. Repo: github.com/pinceladasdaweb/quayside — the docs include the full state machine and the design notes behind everything above.

If you take one thing from this post, take the asymmetry. The next time an event tempts you to release a lock early because holding it is inconvenient: the inconvenience is bounded and visible, the early release is unbounded and silent. Hold.

"half-open" twice is not the same state: the bug that shaped breakwater 1.0

Pedro Rogério — Sat, 15 Aug 2026 17:19:55 +0000

breakwater is my resilience toolkit for Node.js — retry, circuit breaker, timeout, bulkhead, rate limiting, stale-while-open caching, all composable, with observability built in. It just hit 1.0.0, and the headline feature is the one no Node library did well: a circuit breaker whose state is shared across every instance of your service.

One instance sees the outage and trips the breaker. The others fail fast immediately, without each having to discover the same outage on their own users. When the cooldown elapses, exactly one of them probes the recovering dependency while the rest keep waiting.

That is the pitch. This post is about the two things I got wrong on the way there, because they were both the kind of wrong that looks right.

Part 1: a state machine where the same name means two different things

A circuit breaker is a tiny state machine. Closed, open, half-open. In one process you protect transitions with nothing at all — JavaScript is single-threaded, and the code between two awaits cannot be interrupted.

Share that state across N instances and every transition becomes a compare-and-set. That much I knew. So the store interface had:

transition(name, from, to): boolean   // swap only if the state is still `from`

Atomic in Redis via a Lua script. Looks correct. It is not.

Here is the sequence that breaks it. The circuit is half-open and one instance is probing:

A probe fails. The instance decides: reopen the circuit.
That decision travels — a Lua round trip to Redis, a few milliseconds.
In those milliseconds, another probe succeeds, reaches the majority, and closes the circuit. Traffic resumes. It fails again. The circuit reopens, waits out the cooldown, and enters half-open again.
Now the first instance's swap lands. It says: "if the state is still half-open, make it open."

The state is half-open. The swap succeeds. And it is completely wrong — that decision belonged to a period that ended three transitions ago, and it just killed a recovery that had nothing to do with it.

This is the ABA problem, and it is easy to miss here because the states have names. half-open looks like an identity. It is not: it is a label that the circuit wears repeatedly, and comparing labels tells you nothing about whether you are still in the world you made your decision in.

My first fix was a patch, and I knew it

I noticed the race before I had a distributed store — the in-process breaker had the same window whenever a custom store was async. So I patched it: after the swap succeeded, check whether the period had flipped, and if it had, swap the state back.

I even wrote the honest comment:

// The half-open period this failure belonged to ended while the CAS
// travelled: the trip landed on a FRESH period. Hand the state back
// — best effort until stores can fence the CAS with a generation.

"Best effort" is a confession. Two swaps are not one swap: between them, another instance sees the wrong state, and the compensating swap can itself fail. I shipped it because the alternative was redesigning the store contract, and I wasn't ready.

The real fix: identify the period, not the state

The fix is a fence — a monotonic token the store mints on every successful transition:

readState(name): { state, fence, openedAt? }
compareAndSet(name, from, to, fence): { ok, snapshot }

The swap lands only if the state is still from and nothing has transitioned since you read that fence. The stale decision from step 4 now carries fence 7 against a store holding fence 10. Redis refuses it, atomically, in the same script. No compensation. No window.

Both compensating transitions were deleted. The code got shorter, which is how you know the design got better.

Two details that fell out of it, and that I would not have thought to add:

The failed swap returns where the circuit actually is. Losing a race used to cost a second round trip to find out what happened. Now the outcome carries the current snapshot — you lose the race and refresh your view in one shot.

The store owns the timing. openedAt lives in Redis, stamped from the server clock. Before that, an instance that never saw the trip started counting the cooldown from the moment it first noticed — so instances disagreed about when probing was allowed, and the one that noticed earliest probed too soon. Now they all read the same number.

Part 2: a promise that never settles is worse than an error

The whole premise of a distributed circuit breaker is that Redis is now on the path of every protected call. Which raises the obvious question: what happens when Redis is down?

I had an answer I was proud of. No method of the store ever rejects. If Redis is unreachable, the store answers from what the instance already knows and the circuit simply becomes local until Redis comes back. You lose agreement between instances, not the protection itself. A resilience library that fails when its own backend fails has the problem backwards.

I tested it thoroughly. Store throws → contained. Store rejects → contained. Every method, both paths, all green.

Then a reviewer asked what happens when the client doesn't throw or resolve.

ioredis, with default options — the exact configuration my own documentation recommended — has enableOfflineQueue: true. When the connection is down, commands are queued, not rejected. They settle after the retry budget runs out. Measured:

readState against a dead Redis      -> settled after 73151 ms
breaker.execute() with Redis down   -> STILL PENDING after 8000 ms

Seventy-three seconds. And it is worse than it looks, because the degradation logic never engages: the code that decides "Redis is down, go local for 5 seconds" runs in the catch, and nothing ever reaches the catch. Every protected call, on every breaker sharing that store, waits.

My library, whose entire job is to stop a dying dependency from taking your service with it, would have taken your service down with a dying Redis. Not with an error — with silence, which is worse, because nothing times out and nothing gets logged.

The fix is a bound the store owns rather than inherits:

const raw = await Promise.race([
  client.runScript(script, keys, args),
  timeoutAfter(commandTimeoutMs)   // default 500ms
])

Documenting enableOfflineQueue: false would not have been enough. The store's central promise cannot depend on wiring it does not control.

What the pre-1.0 review actually found

I do not ship a release without a review pass, and for 1.0.0 I ran four — core concurrency, the Redis adapter and its Lua, the public API and docs, and security. Between them, seven serious defects, every one reproduced by execution before I touched anything:

A breaker that, with a store that does not report timing, stopped protecting permanently — nextAttemptAt landed in the past and every subsequent call was admitted as a probe, forever.
Probe slots going negative, so more concurrent probes than configured hit a recovering dependency.
The 73-second stall above.
An isolate() kill switch — the one you use to cut off a compromised dependency — that un-isolated itself after a TTL.
A store taken offline by one slow command that failed after a newer command had already succeeded, proving Redis was up.
A float ttlMs that aborted the Lua script mid-write, leaving keys with no expiry at all.
An unvalidated reply that could wedge the circuit open with no way back.

Here is the part I want to underline: several of those were in code I had just written to fix other bugs. The self-un-isolating kill switch was a direct consequence of a TTL fix I had applied an hour earlier — renewing the lease on every read, which quietly undid the PERSIST that made isolation permanent.

I now treat the fix batch as its own reviewable unit, because that is where a meaningful share of my bugs live. Fixing something is when you are most confident and least careful.

And then CI found an eighth, after all 379 local checks passed. My command timeout used an unref'd timer — the kind of tidiness that seems obviously correct. In a process with nothing else pending, the event loop drains, the timer never fires, and the caller's promise never resolves. My protection against hanging had a hanging path of its own. It only shows up on a machine quiet enough to notice.

What 1.0 actually promises

A 1.0.0 that does not say what it protects is just a number. So the release also ships a versioning policy that spells out what semver covers here — and one distinction I had not seen stated elsewhere.

Most types in a library are ones you consume. Adding a field to those is a minor: your code keeps compiling.

But four interfaces exist for you to implement — the state store, the cache store, the metrics collector, the Redis client boundary. For those, the direction reverses: adding a required member is a breaking change, because your implementation suddenly no longer satisfies the interface. So on those four, new capabilities arrive as optional members, and a member only becomes required in a major.

That rule cost one member its place before the freeze. StateStore.subscribe was declared for a push-based invalidation that nothing called yet. An optional method the library never invokes is a promise you are not keeping — somebody implements it and waits for calls that never come. It came out. It returns when there is code behind it.

Try it

npm install breakwater ioredis

import Redis from 'ioredis'
import { circuitBreaker } from 'breakwater'
import { redisStore, fromIoredis } from 'breakwater/redis'

const client = new Redis(process.env.REDIS_URL ?? 'redis://127.0.0.1:6379')
const store = redisStore({ client: fromIoredis(client) })

const payments = circuitBreaker({
  name: 'payments-api',   // the key the circuit is shared under
  stateStore: store,
  failureThreshold: 0.5,
  minimumCalls: 20
})

const receipt = await payments.execute(({ signal }) =>
  api.post('/charge', body, { signal })
)

That is the whole integration. Composition, events, and the Prometheus and OpenTelemetry adapters work exactly as before — only where the state lives has changed.

GitHub: pinceladasdaweb/breakwater
npm: breakwater
The Redis page, including everything a shared circuit cannot promise: docs/redis.md

If you are building anything that shares state across instances, the lesson worth stealing is the first one: a compare-and-set on a value that repeats is not a compare-and-set. Fence it, or you are trusting a name to be an identity.

My test suite had 100% coverage. Mutation testing still found real bugs.

Pedro Rogério — Mon, 03 Aug 2026 12:36:49 +0000

breakwater is my resilience toolkit for Node.js — retry, circuit breaker, timeout, bulkhead, rate limiting, all composable, with observability built in. It's the library that stands between your service and a dying dependency, which means its tests are not decoration. If the circuit breaker's threshold logic is wrong, somebody's incident gets worse.

The suite had 100% line coverage on the core. Every branch tool was green. I was feeling pretty good about it.

Then I ran Stryker and it graded my suite at 80.52%.

Coverage measures execution. Mutation measures assertion.

If you haven't used mutation testing: the tool takes your source, makes one small change at a time — flips a >= to >, replaces a condition with true, empties a block — and runs your suite against each mutant. If some test fails, the mutant is killed. If everything stays green, it survived: your tests executed that line and asserted nothing about it.

Coverage tells you the line ran. Mutation testing tells you whether anyone would notice if the line were wrong. Those are very different claims, and the gap between them is exactly where production bugs live.

Stryker generated 1,114 mutants from my source. 199 survived.

What the survivors actually were

I went through every single one. They fell into five buckets, and each bucket taught me something different.

1. A real bug, in production, that day

A cluster of survivors pointed at the code that injects a registry name into policy options. Writing the test that should have killed them uncovered an actual defect: a named policy entry with an explicit display name reported the pipeline under one name and its rate limiter under another — the same policy showing up as two series in a metrics dashboard, in direct contradiction of the documented "explicit names always win" rule.

That fix shipped to npm the same week. No user had reported it. Coverage had been green the whole time.

2. A test that was lying about what it tested

One test existed specifically to pin a floating-point rounding fix: retryAfterMs must always be sufficient — wait exactly that long and you're admitted. Its comment proudly said the configuration reproduced a case where the naive ceil() lands one millisecond short.

Except it didn't anymore. The configuration (interval: 8737) no longer triggered the correction loop at all — the test passed with or without the code it claimed to protect. I searched the parameter space for a configuration that actually hits the rounding error (interval: 161, if you're curious: 161 * (1/161) is 0.999…) and rewrote the test around it.

A test that asserts something true but unrelated to its purpose is worse than no test: it's documentation that lies.

3. Documented contracts nobody ever asserted

The biggest bucket. Things the README and docs promised, that all worked, and that no test would defend against regression:

The circuit breaker's open, close and halfOpen events — only the generic stateChange was ever checked. Payloads, correlation IDs, one-event-per-transition: all unpinned.
Every distributed state-store path. The breaker has a pluggable StateStore designed for sharing circuit state across instances — stores that answer asynchronously, losing a compare-and-set race to a peer, another instance winning the half-open probe election. The interface existed, the local implementation was tested, and not one distributed behavior had a test.
Boundary semantics: a failure rate landing exactly on the threshold, a retry delay landing exactly on the deadline.
Jitter that actually spreads. My tests asserted delays stayed within [0, max] — a jitter implementation returning a constant would have passed while doing nothing about thundering herds.
A metrics collector implementing none of its optional callbacks. This one hid behind deliberate error-swallowing (monitoring must never break execution), so the fix asserts the error reporter stays silent too.

4. Dead code — the report doubles as a detector

Some survivors survived because no input can reach them. Two examples from the timeout policy: a guard in the timer callback protecting an assignment that nothing could ever read (the catch block re-checks the same condition first, and signals never un-abort), and one arm of a condition that was provably always false at its only call site.

I didn't take "provably" on faith — I rebuilt the old version and ran a differential matrix across cooperative/aggressive modes, abort orderings and rejection identities. Zero divergence. Both paths deleted. The invariant they pretended to guard now lives in exactly one place, with a test pinning its sharpest corner: an external cancellation that lands after the deadline fired still wins, and the timeout event stays silent.

5. Equivalent mutants — know when to stop

Not everything should be killed. { once: true } on an abort listener is unkillable by any black-box test, because an AbortSignal fires abort at most once in its lifetime — the mutant is behaviorally identical. Same for a fast-path that skips building a composite signal: pure allocation optimization.

Chasing those means asserting listener counts and internal allocations — testing implementation, which is the same disease as coverage-theater in the opposite direction. I documented each accepted survivor and moved on. The honest end state isn't 100%: it's every survivor analyzed and either killed or justified.

Final score: 95.11%, with the suite growing from 157 to 223 tests. And one bonus find that wasn't a mutant at all: two of my new tests would have hung CI forever instead of failing if their behavior regressed, because node:test has no default timeout. If you use the built-in runner, set one:

"test": "node --import tsx --test --test-timeout=10000 'tests/**/*.test.ts'"

The setup, if you want this on node:test

Stryker's tap runner drives the native runner just fine, TypeScript included:

{
  "testRunner": "tap",
  "tap": {
    "testFiles": ["tests/**/*.test.ts"],
    "nodeArgs": ["--import", "tsx", "--test", "--test-reporter=tap", "--experimental-test-isolation=none"]
  },
  "mutate": ["src/**/*.ts"],
  "coverageAnalysis": "perTest",
  "reporters": ["json", "html", "progress"],
  "timeoutMS": 20000
}

coverageAnalysis: "perTest" matters — it runs only the tests covering each mutant, which took a full run of my suite to under six minutes. I keep it as a manual gate (npm run test:mutation) rather than a CI job: it's a code-review tool, not a merge blocker.

Why I did all this before shipping observability

Because the newest release of breakwater is an observability feature, and shipping a tool that watches production on top of tests I couldn't trust felt absurd.

breakwater@0.7.0 adds breakwater/prometheus — ready-made prom-client collectors for every signal the library emits:

import { resilience } from 'breakwater'
import { prometheusCollector } from 'breakwater/prometheus'

const payments = resilience({
  name: 'payments-api',
  retry: { attempts: 3 },
  circuitBreaker: { consecutiveFailures: 5 },
  timeout: 2_000,
  metrics: prometheusCollector()
})

That's the whole integration. Eight metrics come out: executions and a duration histogram by outcome, retries, timeouts, fallbacks, rejections by reason (circuit_open, bulkhead_full, rate_limited…), the circuit state as an enum gauge, and a transitions counter. prom-client is an optional peer dependency — the core keeps its zero runtime dependencies, and importing plain breakwater never loads it.

Two design notes that came straight out of caring about honest observability:

Labels are low-cardinality by construction. Correlation IDs and attempt numbers never become label values. The only free-form label is your policy's name.

Absence means healthy. The state gauge is event-driven, so a circuit that has never tripped exports no state series yet. The docs say so explicitly, and the suggested alert targets the gauge (breakwater_circuit_state{state="open"} == 1) instead of increase() on a transitions counter — a counter series born by its first-ever transition needs two samples before increase() sees anything, which is exactly how you miss your first incident.

There's a ready-to-import Grafana dashboard covering all eight metrics, and a runnable demo — docker compose up brings up a breakwater-protected app with a scripted outage every 90 seconds, Prometheus scraping it, and Grafana already provisioned. You get to watch the circuit open, fallbacks take over, and a half-open probe close the loop, on a real dashboard, without wiring anything.

And yes: the adapter's module was born under the mutation gate. First report: 100% — zero survivors. It turns out writing honest tests is much easier when you do it from the start.

Takeaways

Coverage is a floor, not a verdict. 100% coverage coexisted with 199 unasserted behaviors in my suite.
Read every survivor. The value isn't the score — it's the taxonomy: real bugs, lying tests, unpinned contracts, dead code, and equivalents you consciously accept.
Mutation reports find dead code that no linter flags, because reachability is a semantic property.
Set a test timeout with node:test. A test that hangs instead of failing is a CI outage waiting for a regression.
Don't chase 100%. Killing equivalent mutants means testing internals. Analyze, justify, stop.

breakwater is on GitHub and npm — MIT, docs for every policy, and the comparison with opossum/cockatiel is in the README. If you run it behind Prometheus, I'd genuinely like to hear what your dashboards catch.

Node.js has plenty of circuit breakers. So why did I build another one?

Pedro Rogério — Mon, 27 Jul 2026 12:40:00 +0000

Every service I've worked on eventually grows the same scar tissue: a retry loop copy-pasted into six files, a circuit breaker bolted onto the payment client after an outage, a timeout wrapper someone wrote at 3 a.m. Each one slightly different. None of them talking to each other. And when things go wrong, nobody can answer the only question that matters during an incident: what is the resilience layer actually doing right now?

Java solved this years ago with resilience4j. .NET has Polly. Node.js... has pieces.

The gap

I evaluated what the ecosystem offers before writing a single line:

opossum is the best-known circuit breaker, mature and well maintained. But it's only a circuit breaker — retry is rudimentary, there's no bulkhead, no composition. Metrics need a plugin.

cockatiel is the closest thing to Polly: retry, breaker, timeout, bulkhead, composition. I genuinely like its design. But observability is where it stops — no native metrics, no pipeline-wide correlation — and maintenance has slowed.

The Sindre micro-libs (p-retry, p-timeout, p-limit) are excellent at exactly one thing each. But resilience is a system: a retry that doesn't know the circuit is open will happily sleep through backoff to hammer a dead dependency. Isolated pieces can't coordinate.

And there was one thing nobody documented properly, which became the reason I finally started typing:

Ordering is the whole game

Take four policies: retry, circuit breaker, timeout, fallback. The same four, nested in two different orders, produce two very different systems:

retry( circuitBreaker( timeout( fn ) ) )   // A
circuitBreaker( retry( timeout( fn ) ) )   // B

In A, every attempt flows through the breaker, so the breaker sees the dependency's true failure rate — and when the circuit opens mid-retry, the retry finds out immediately.

In B, the breaker sees one outcome per retry cycle: three real failures against the dependency count as a single failure. The circuit opens far later than the dependency's actual state justifies, and the retry keeps sleeping through backoff against a broker that's clearly down.

Most libraries let you build either one without ever telling you there's a difference. That's not an API problem — it's a documentation-of-consequences problem. I wanted a library where composition and its consequences are the headline feature, not an afterthought.

So I built breakwater.

What it looks like

npm install breakwater

import { resilience, exponential } from 'breakwater'

const payments = resilience({
  retry: { attempts: 3, backoff: exponential({ initial: 200 }) },
  rateLimit: { limit: 100, interval: 60_000 },
  bulkhead: { concurrency: 20, queue: 50 },
  circuitBreaker: { name: 'payments-api', failureThreshold: 0.5 },
  timeout: 2_000,
  fallback: () => ({ status: 'pending', queued: true })
})

const charge = await payments.execute(({ signal }) => api.post('/charge', body, { signal }))

resilience() is the batteries-included path with a fixed, documented order:

fallback( retry( rateLimit( bulkhead( circuitBreaker( timeout( fn ) ) ) ) ) )

Each position is deliberate. The local guards (rate limit, bulkhead) sit outside the breaker, because your own saturation must never open a circuit that describes the dependency's health. The timeout sits innermost so every attempt gets its own budget — and a hung call becomes a countable failure instead of an invisible one.

Need a different order? compose() reads exactly like the nested calls:

import { compose, retry, circuitBreaker, timeout } from 'breakwater'

const policy = compose(
  retry({ attempts: 3 }),          // outermost
  circuitBreaker({ name: 'api' }),
  timeout(2_000)                   // innermost
)

And because the result of compose() is itself a policy, compositions compose again. The ordering guide walks through the classic configurations with sequence diagrams — it's the page I wish every resilience library had.

Details that only show up in production

A few behaviors took far more design effort than any diagram suggests, because they only matter when things are already going wrong:

Policies coordinate through errors. Every error carries a stable code and a retryable flag. CircuitOpenError declares itself retryable: false — so the default retry gives up instantly instead of backing off against an open circuit. Rate-limit and bulkhead rejections stay retryable, because saturation is transient. This is why the pieces compose correctly out of the box.

Cancellation is not failure. One AbortSignal — combining your external cancellation, timeouts, everything — reaches your function. And when the caller aborts, nothing counts it as a failure: retry doesn't retry it, the breaker doesn't tally it, fallback doesn't replace it. Your user closing a tab should never open a circuit.

Timeouts don't lie. A genuine domain error that lands after the deadline propagates as itself, never masked as a TimeoutError. A call the caller cancelled is cancellation, not a timeout. Your error dashboards reflect what actually happened.

Monitoring never changes an outcome. An event listener or metrics collector that throws is isolated and reported — a bug in your telemetry cannot turn a successful payment into an error.

Observability without plugins

Every policy emits typed events, one correlationId crosses the whole pipeline, and stateful policies expose stats():

breaker.stats()
// { state: 'open', failureRate: 0.85, lastError, openedAt, nextAttemptAt, ... }

CircuitOpenError carries that snapshot too — a proper Retry-After header is three lines. For metrics pipelines, implement one MetricsCollector interface and it wires everywhere; define your policies once in a named registry and every metric comes out labeled:

import { policies } from 'breakwater'

policies.define('payments-api', { retry: { attempts: 3 }, circuitBreaker: {}, timeout: 2_000 })

// anywhere else — same name, same instance, genuinely shared circuit state
await policies.get('payments-api').execute(({ signal }) => api.post('/charge', body, { signal }))

A health endpoint over every policy in your app is a policies.names().map(...) away.

Dogfooding, honestly

I maintain a RabbitMQ client library that had its own hand-rolled circuit breaker and retry loop — the exact scar tissue this post opened with. Migrating it to breakwater deleted the bespoke resilience code, upgraded the publisher to the correct retry-outside-the-breaker ordering, and every one of its existing tests passed unchanged, including destructive reconnection tests against a real broker. That migration shipped; it's running in production today.

It also taught me things no unit test would: that consumers of a hand-rolled retry expect the raw last error (there's a documented unwrap pattern now), and that sync call sites need a recreate-the-policy idiom for breaker resets. Both learnings went straight into the docs.

What it is — and isn't (yet)

breakwater today: retry (backoff strategies + jitter + total deadline), timeout (cooperative and aggressive), circuit breaker (sliding windows, bounded half-open probing, manual isolation), bulkhead, rate limiting (token bucket and exact sliding window), chained fallbacks, composition, typed events, metrics, named policies. Zero runtime dependencies, TypeScript-native, dual ESM/CJS, Node >= 22.

Planned and designed for (the circuit breaker's state store is already pluggable): distributed circuit breaker state over Redis — ten instances of your service agreeing that an endpoint is down, with a single instance elected to probe recovery. Plus ready-made Prometheus/OpenTelemetry collectors and a stale-while-open response cache.

The API is heading to 1.0; until then, minor versions may adjust it.

Try it

GitHub: github.com/pinceladasdaweb/breakwater
npm: npmjs.com/package/breakwater
Start here: the composition & ordering guide

If you've ever debugged a retry storm at 3 a.m., I'd genuinely love your feedback — issues and PRs welcome. And if breakwater saves you from writing one more bespoke circuit breaker, it did its job.

My Redis library said the write succeeded. Redis was down.

Pedro Rogério — Fri, 24 Jul 2026 12:59:40 +0000

Last year I audited my RabbitMQ library and discovered it couldn't survive a single reconnect. I turned the whole ordeal into a playbook, and I promised myself I'd run it against the next library in my drawer.

The next library was a Redis client I wrote two years ago. The README advertised connection pooling, pub/sub, distributed locking, "battle-tested error handling" and "production-ready resilience".

Here's what the audit found: there was no pooling. There was no pub/sub. There was no locking. There were no tests. And the resilience — the one thing a Redis wrapper exists for — was worse than absent. It was actively lying.

This is the story of the second time I learned the same lesson, plus the parts that were new.

The write that succeeded by doing nothing

Every command in the library went through a health check. If Redis was unhealthy, this happened:

if (!isHealthy) {
  this.logger.warn(`Redis is not healthy. Skipping ${command} operation.`)
  return null
}

Read that again. set, incr, lpush — writes — would resolve successfully having done nothing. No throw, no rejection. Your checkout flow calls set, gets a resolved promise, tells the user their cart is saved. The cart does not exist.

The best part? The README documented this as a feature: "Methods return null when Redis is not healthy." I had documented data loss and called it graceful degradation.

And for reads it's worse in a subtler way: get returning null means "key doesn't exist" — or now, "Redis was down". Your cache layer can't tell a miss from an outage.

The fix is a contract, not a patch:

throw new RedisClientError(
  `Redis is not connected. Cannot execute '${command}'.`,
  command,
  'REDIS_UNAVAILABLE'
)

Structured errors with a code. The caller branches on err.code === 'REDIS_UNAVAILABLE', never on message text. A write that can't happen must be loud. This was a breaking change — and it's exactly why I fixed it before the library had public users. Defaults are free to change until the day someone depends on them.

The reconnection layer that fought the driver — and lost to itself

My library sits on top of ioredis. ioredis already reconnects. It has retryStrategy, exponential backoff, an offline queue, resubscription. It is good at this.

Two-years-ago me didn't trust it, so he built a second reconnection layer on top. Every error or close event scheduled a manual reconnect:

scheduleReconnect () {
  this.reconnectAttempts++
  setTimeout(() => this.attemptConnection(), this.reconnectInterval)
}

async attemptConnection () {
  this.client = this.redisConfig.createRedisClient()  // ← a NEW client
  // ...
}

Spot it? Every attempt created a brand-new ioredis client without destroying the previous one. The old client — with its own retryStrategy — kept reconnecting forever in the background. Every drop leaked one more client, each running its own infinite retry loop against your server. A one-minute Redis outage turned my "resilience layer" into a connection storm.

It gets better. After the first attempt, the validation ping was skipped entirely:

if (!this.initialConnectionAttempted) {
  await this.client.ping()
  this.initialConnectionAttempted = true
}
// every reconnection after this point:
this.isConnected = true   // sure, why not

Every reconnection "succeeded" unconditionally. isConnected = true while Redis was on fire. The reconnection had never worked and never been tested — the exact disease from the RabbitMQ library, in a different body.

And my favorite: disconnect() called quit(), which made ioredis emit close… and the close handler scheduled a reconnect. Graceful shutdown resurrected the connection. The library refused to die. There's something poetic about a resilience layer whose only reliably working feature is refusing to let you leave.

The fix was not code. It was deletion. The entire manual layer went away — the driver is the single owner of reconnection, and my library only tracks state and emits events. Four critical bugs died in one commit that removed more lines than it added.

The test that earns the word "resilient"

From the RabbitMQ refactor I kept one ritual: no reliability claim without a test that kills the connection for real, from the server side. For Redis, that's CLIENT KILL:

await client.set('sacred:before', 'ok')          // works
await killAllClientsFromTheServer()               // CLIENT KILL, server-side
await waitFor(async () =>
  await client.set('sacred:after', 'ok') === 'OK' // works again, same client
)

Plus the two regression probes that document the old bugs forever: after recovery, the server must see exactly the same number of connections as before (no leaked reconnection loops), and after disconnect() the client must stay disconnected (no resurrection).

First run against the old code: the resurrection probe failed exactly as predicted, and the test runner never exited — leaked timers kept the process alive. The suite needed Ctrl-C. When your test suite can't die, your library can't either.

After the surgery: everything green, the runner exits by itself, and recovery after a server-side kill went from ~2 seconds to ~110 milliseconds — because reacting to the driver's events beats waiting for your own arbitrary timer.

A parade of small lies

The deep read found plenty more. A sample:

The scan that read other people's keys. getAllStream() used ioredis' scanStream — which does not apply your keyPrefix to the MATCH pattern. With a prefix configured, scanning for user:* matched zero of your own keys. And scanning * swept the entire database, including other applications' keys, which then got your prefix wrongly added on the follow-up GET and came back as null. The old code had mysterious logging for these "null values", fetching TTL and type for each one. That wasn't observability. That was a bug filing reports about itself.

maxRetryAttempts: 0 meant infinity. The classic options.x || default — zero is falsy, so "never retry" became "retry forever". Same for block: 0 in stream reads, which in Redis means "block forever" and was silently dropped. ?? exists. Use it.

XGROUP DESTROY couldn't work. The wrapper appended a default stream id to every subcommand, so the destroy documented in my own README sent a stray $ and died with a protocol error. The README showed example code that had never once executed successfully.

Every command paid a toll. The health check ran a real PING — serialized before your command, up to once per five seconds. Fail-fast checks must be local and free (client.status === 'ready'); the driver already owns the expensive part.

None of these are exotic. They're what the first test you ever write catches. There were no tests.

Then I had to earn the README

Deleting fiction from the README was easy. But two of the fictional features were genuinely good ideas — so this time they got built for real, with the reliability ritual applied from day one.

Pub/sub lives on a dedicated connection (a subscribed Redis connection can't run normal commands — the library manages that for you), and its sacred test kills the subscriber server-side and proves messages flow again after recovery.

Distributed locking is SET NX PX with a random token, and release/extend go through Lua scripts that check the token — a holder whose lock expired can never delete the new holder's lock. The README says, in bold, that this is a single-instance mutex and not Redlock. Honesty is a feature.

And they compose. The cache helper uses the lock internally for stampede protection:

const report = await redis.getOrSetJson('report:daily', 3600, buildReport, { lock: true })

Ten concurrent cache misses, one producer call — the test proves exactly that, with real concurrency against a real Redis.

The reviews that found what I couldn't

"Looking finished" is still the most dangerous state, so finished code got adversarial reviews — every finding had to be reproduced live before I accepted it. Two highlights that survived the process:

A producer returning undefined in the cache helper poisoned the key: JSON.stringify(undefined) is undefined, which got stored as an empty string with a TTL — and every later read exploded with SyntaxError until it expired. One absent-minded return and the key is bricked for an hour.

And the packaging check — install the actual tarball in a clean project — caught two defects no unit test could: my prepare script made npm v12 nag every consumer with an allow-scripts approval warning, and TypeScript consumers in CommonJS hit TS1479 because a "type": "module" package needs dual declaration files (index.d.ts + index.d.cts). Validate the artifact, not the repo.

What shipped

50 tests: 24 unit, 26 integration against a real Redis — including server-side CLIENT KILL recovery for both commands and pub/sub
1 runtime dependency (ioredis). The old version had three: pino and pino-pretty shipped a dev log formatter to every production install. Bring your own logger; the fallback is 25 dependency-free lines
Fail-fast structured errors, observable lifecycle events, working optimistic locking via dedicated connections
CI on Node 22/24 with a real Redis service container, npm trusted publishing (OIDC) with provenance, zero lifecycle scripts in the package

It's on npm as @pinceladasdaweb/redis, source at github.com/pinceladasdaweb/redis.

The lessons, second edition

Docs are a spec you've already published. Either the code honors them or the docs change. My README described a different, better library — and the gap was invisible because nothing tested the claims.
Read everything before fixing anything. The four worst bugs looked independent and were one defect: a reconnection layer that shouldn't exist. Patch-by-patch would have polished each symptom and kept the disease.
Don't own what your driver owns. The best resilience code I wrote this month was a deletion. If you're wrapping a client that already reconnects, your job is what happens around reconnection — state, events, dedicated connections — not a competing loop.
A write that resolves while doing nothing is data loss with good manners. Fail fast, fail loud, fail with a code.
Kill it from the server side. CLIENT KILL is one command. If your library claims resilience and your test suite has never dropped a connection for real, you don't have a resilience feature — you have a resilience rumor.

Same ending as last time, because it's still true: this library now survives everything I could think of throwing at it, and it has zero production mileage under other people's workloads. If you run Redis with Node and feel like breaking something, I'd genuinely love the bug reports.

Turns out "looking finished" is still the bug. At least this time I knew where to look.

My RabbitMQ library looked production-ready. It couldn't survive a single reconnect.

Pedro Rogério — Sun, 19 Jul 2026 17:13:46 +0000

For a couple of years I had a RabbitMQ client library sitting in a private repo. On paper it was impressive: automatic reconnection, channel pooling, a circuit breaker, four rate-limiting strategies, gzip compression, dead letter queues, worker-thread consumers, delayed messages. Over 1,200 lines in the main class alone.

I never quite trusted it enough to publish. Recently I decided to find out whether that instinct was right.

It was. The single most important feature of a message-broker client — surviving a connection drop — had never worked. Not "worked poorly". Never worked. And nothing in the repo could have told me, because there wasn't a single test.

This is the story of the audit, the rebuild, and the release of @pinceladasdaweb/rabbitmq. It's also a collection of lessons I wish I'd internalized years earlier, because none of these bugs were exotic. They were all sitting in plain sight, waiting for a failure path to be exercised for the first time in production.

The audit: what "looks done" actually hides

The library had the shape of a finished product. Rich API, detailed README, twenty example scripts. Here's a sample of what a deep read of the code actually found.

The reconnection that reconnected to nothing. The connection layer dutifully detected drops, retried with exponential backoff, and re-established the TCP connection. Then... nothing. The channel pool — the thing every publish and consume goes through — was only ever created inside connect(), and the reconnection path never called it. After any network blip, every operation would throw Not connected forever, while the connection state proudly reported connected. The recovery code existed; it recreated consumers using channels captured in closures from before the drop — channels that were dead. It had clearly never been executed.

The rate limiter that crashed on its own feature. The block-a-key feature stored blocked keys in a Set... and called this.blocked.set(key, Date.now()) on it. Set has no .set(). Calling blockKey() threw a TypeError, every time, since the day it was written.

The circuit breaker state that didn't exist. The public API exposed getCircuitBreakerState(), which called this.#circuitBreaker.getState() — a method the circuit breaker class simply did not have.

The dead-letter helper with the wrong signature. moveToDeadLetter() called the publish method with four arguments in the wrong order, so the name of the queue was published as the message body, to the wrong exchange.

The library that killed your process. The constructor registered SIGINT, SIGTERM, uncaughtException and unhandledRejection handlers — and called process.exit() in them. A library. Any unhandled rejection anywhere in your application would cause my dependency to shut your process down.

None of these are subtle bugs. They're the kind of thing the first test you ever write catches. Which brings me to the first lesson.

Lesson 1: features are cheap; invariants are expensive. An API surface can look complete while every failure path is fiction. The happy path is maybe 20% of an infrastructure library — the rest is what happens during the bad minutes, and that's exactly the part that "it runs on my machine" never exercises.

If you claim resilience, your CI has to actually break things

The rebuild started with the reliability core: reconnection has to restore the channel pool, re-assert topology, and recreate every consumer on fresh channels. But I'd been burned by "code that looks like it recovers", so the real work was making the claim falsifiable.

The integration suite now runs against a real RabbitMQ (Docker locally, a service container in GitHub Actions) and includes this test — the one I care most about in the entire repo:

test('recovers channel pool and consumers after a forced connection drop', async () => {
  const reconnected = new Promise(resolve => rabbitMQ.once('reconnected', resolve))

  // Ask the broker to close every AMQP connection, simulating a real outage.
  await forceCloseAllConnections() // DELETE /api/connections/* via the management API

  await reconnected

  const countBefore = received.length
  await rabbitMQ.publish(ROUTING_KEY, { after: 'reconnect' })

  await waitFor(() => received.length > countBefore)
  assert.equal(rabbitMQ.getClusterStatus().connectionState, 'connected')
})

No mocks. The broker genuinely kills the connection; the library genuinely has to come back: pool rebuilt, exchange re-asserted, consumer re-subscribed on a new channel, message flowing again. This runs on every pull request.

Using the management HTTP API for the kill switch (instead of docker restart) was a small trick that paid off: it works identically on a laptop and inside a CI service container, and it's fast.

Lesson 2: a resilience feature without a test that injects the failure is a rumor. The test that kills your broker connection is worth more than the thousand lines of recovery code it validates.

The adversarial review: 42 candidates, 10 confirmed bugs

With the core rebuilt and a test harness in place, I ran a structured review of the source: multiple independent passes over the code, each hunting from a different angle — line-by-line, lifecycle invariants, cross-module contracts, plus efficiency and design-depth sweeps. Every candidate finding then got adversarially verified against the code before I accepted it.

Ten confirmed bugs came out. Three of my favorites, because each one represents a category:

1. The option that silently discarded messages.

async retryOperation (operation, maxRetries = 3, delay = 1000) {
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    // ...
  }
}

Spot it? Pass { maxRetries: 0 } — a perfectly reasonable "try once, don't retry" intent — and the loop body never runs. The operation is never attempted, channel.publish is never called, and the promise resolves successfully. The circuit breaker even records a success. Silent message loss, triggered by a documented option. The fix is one line (Math.max(1, maxRetries)), plus a regression test that pins the semantics forever.

2. The poison message that ate a consumer. A message with a corrupt gzip body would fail to decode, get nacked with requeue: true, be redelivered instantly, fail again... forever. With prefetch: 1, one bad message parked an entire queue while burning 100% CPU. Undecodable messages now go to the dead letter queue — failure handling should drain problems, not recirculate them.

3. The unroutable dead-letter that vanished. Moving a message to a DLQ published it without the mandatory flag. If the dead-letter routing had no binding — one topology mistake away — the broker would confirm the publish and silently drop the message. The method resolved, logged success, and the message ceased to exist. It now publishes with mandatory: true, listens for basic.return, and rejects loudly when the DLQ isn't routable.

There were seven more: a disconnect()/connect() cycle that permanently disabled auto-reconnection, consumer callback failures tripping the circuit breaker that gates publishing, a fire-and-forget publish that could hang forever awaiting 'drain' on a dead channel, duplicate consumers spawned by racing recovery paths...

Lesson 3: bugs cluster in the seams. Almost every confirmed finding lived at an intersection — between two recovery mechanisms, between an option's edge value and a default, between "the callback succeeded" and "the ack was delivered". Reviews that only read functions in isolation don't find these. You have to trace the interactions.

Your test suite also protects you from everyone else

Two things happened during this project that had nothing to do with my code, and the test suite caught both.

The dependency major-bump. Mid-rebuild, I upgraded amqplib from 0.10 to 2.0 — a jump across two majors of the library this whole thing wraps. The entire integration suite (confirms, compression round-trips, forced reconnections, delayed messages) passed against the new version in one afternoon. Without the suite, that upgrade would have been an act of faith deferred indefinitely.

The floating Docker tag. My CI used rabbitmq:4-management-alpine. One day that tag started resolving to RabbitMQ 4.3, which rejected the delayed-message plugin version I bundled (built for 4.0.x). CI went red before any user ever saw the problem. The fix taught me to pin brokers and plugins as a matched pair (rabbitmq:4.2-management-alpine + plugin 4.2.0) — a floating major tag in CI is a time bomb with a pleasant default.

Lesson 4: automated tests are not just about your bugs. They're your early-warning system for upstream majors, ecosystem drift, and infrastructure that changes underneath you.

Dependencies are promises

At the start, the library shipped four runtime dependencies, including pino and pino-pretty — a pretty-printer for development logs, in every production install.

It now ships two: amqplib and a small cache. The logger question is the interesting one. The library used to create its own pino instance at import time — its own format, its own destination, disconnected from whatever the host application does. But an infrastructure library shouldn't own a logging pipeline; it should write to whatever you give it:

import pino from 'pino'

const rabbit = new RabbitMQ({
  endpoints: ['localhost:5672'],
  logger: pino({ name: 'rabbitmq' }) // any pino/winston/bunyan instance works
})

If you inject nothing, a 25-line dependency-free console logger provides timestamped, leveled output. Nobody pays for a logging stack they already have.

The same thinking applied to process lifecycle: the library no longer touches your signal handlers. Graceful shutdown is one explicit call — rabbit.enableGracefulShutdown() — and it's opt-in, because your process belongs to you, not to your dependencies.

A small release-engineering war story

The publish pipeline (GitHub Actions → npm on every merge to main) had its own baptism by fire. Two merges landed on main in quick succession; two release runs raced each other. The first published 1.0.1 to npm, then failed pushing the git tag back (the branch had moved). The second run recomputed "next version = 1.0.1" from the now-stale package.json — and npm refused to publish over an existing version. Deadlock: every subsequent release run would fail the same way.

Two structural fixes, both worth stealing:

The registry is the source of truth. The workflow now asks npm what the latest published version is (npm view <pkg> version) and bumps from that, never from package.json. Whatever half-finished state a previous run left behind, the next run self-corrects.
Serialize your releases. A concurrency: { group: publish } block means rapid merges queue instead of racing.

Lesson 5: release automation fails in the middle, not at the edges. Design every step to be safe to re-run, and derive state from systems of record, not from files a previous failed run may or may not have updated.

Where it landed

The library is now public: @pinceladasdaweb/rabbitmq.

npm install @pinceladasdaweb/rabbitmq

import RabbitMQ from '@pinceladasdaweb/rabbitmq'

const rabbit = new RabbitMQ({
  endpoints: ['localhost:5672'],
  exchange: { name: 'orders', type: 'direct' }
})

await rabbit.connect()

await rabbit.subscribe('order-processing', async (content, message) => {
  await handleOrder(content) // auto-ack on success, nack → DLQ on throw
})

await rabbit.publish('order.created', { id: 42 }) // publisher-confirmed

rabbit.on('reconnected', () => console.log('survived an outage, consumers restored'))

What's in the box: automatic reconnection with full state recovery (pool, topology, consumers), publisher confirms on everything, per-key rate limiting (token bucket, leaky bucket, fixed and sliding windows), a circuit breaker that's isolated from consumer failures, transparent gzip compression, first-class dead-letter support (createQueue, moveToDeadLetter, processDeadLetterQueue), delayed messages via the official plugin, worker-thread parallel consumers with automatic respawn, dependency-ordered sequential processing, consumer lifecycle management (unsubscribe, consumerCancelled/consumerRecovered/consumerLost events), and TypeScript declarations checked in strict mode on CI.

Guarding all of it: 87 automated tests — 72 unit, 15 integration against a real broker including the forced-outage scenario — running on Node 22 and 24 on every pull request, plus 22 runnable examples that double as living documentation.

And in the spirit of the whole post, the honest part: this library has excellent test coverage and zero production mileage under other people's workloads. Those are different kinds of confidence, and only one of them can be built alone. If you run RabbitMQ in Node and any of this resonates, I'd genuinely love an issue, a hard question, or a war story from your queue topology: github.com/pinceladasdaweb/rabbitmq.

The library I was too embarrassed to publish taught me more in this rebuild than it did in all the years it sat in a folder looking finished. Turns out "looking finished" was the bug.

Node URL Shortener

Pedro Rogério — Fri, 11 Jun 2021 19:49:08 +0000

I recently ended up creating a URL shortener using NodeJS, Fastify, Postgres and Redis for study purposes. If you think it might be useful for you, if only for study purposes too, come take a look:

https://github.com/pinceladasdaweb/node-url-shortener

Developing a RESTful API using Fastify

Pedro Rogério — Mon, 08 Feb 2021 10:58:57 +0000

You who are tired of seeing examples of REST API in Node using Express will be very interested in this boilerplate that I ended up developing using Traefik, Docker, Docker Compose, Fastify, JWT and Mongodb.

https://github.com/pinceladasdaweb/docker-fastify-restful-api

Happy coding!