How to Stop Cascading Failures with @nestjs-redisx/circuit-breaker

Stop Cascading Failures Across Every Instance

Got a downstream dependency that's acting up — maybe a payment gateway, a search cluster, or some partner API? The knee-jerk reaction is to keep hammering it with requests. But here's the rub: each request sits there, waiting to time out, hogging a connection, and then failing anyway. Under heavy load, this is how one sluggish dependency can bring down your entire service — connection pools run dry, event loop latency shoots through the roof, and even the healthy endpoints start choking.

The circuit breaker pattern is your savior here. It fails fast, cutting the problem off at the knees. After a set number of failures, the breaker trips, and any following calls get an immediate answer — no timeouts, no held connections — giving the troubled dependency a breather.

@nestjs-redisx/circuit-breaker puts this pattern into action. But here's the twist: the breaker state is stored in Redis, shared across every instance of your app.

Why Distributed State Matters

Inline circuit breakers are per-instance, meaning if you've got ten pods, you've got ten independent breakers. Each one learns the failure lesson separately. Set the failure threshold to 5, and the dependency can absorb up to 50 failing requests before all breakers trip. Scale to fifty pods? That's 250.

And recovery? Just as chaotic. Each instance probes independently, so a recovering dependency might face fifty simultaneous probe requests instead of one. Enter Redis-backed state: all instances share a single circuit. Five failures trip the breaker for everyone, and when the cooldown ends, only a limited number of probes get through, not one per instance.

Installation

npm install @nestjs-redisx/core @nestjs-redisx/circuit-breaker ioredis

Configuration

import { Module } from '@nestjs/common';
import { ConfigModule, ConfigService } from '@nestjs/config';
import { RedisModule } from '@nestjs-redisx/core';
import { CircuitBreakerPlugin } from '@nestjs-redisx/circuit-breaker';

@Module({
  imports: [
    ConfigModule.forRoot({ isGlobal: true }),
    RedisModule.forRootAsync({
      imports: [ConfigModule],
      inject: [ConfigService],
      plugins: [
        CircuitBreakerPlugin.registerAsync({
          imports: [ConfigModule],
          inject: [ConfigService],
          useFactory: (config: ConfigService) => ({
            failureThreshold: config.get('CB_FAILURE_THRESHOLD', 5),
            windowMs: config.get('CB_WINDOW_MS', 10000),
            openDurationMs: config.get('CB_OPEN_DURATION_MS', 30000),
          }),
        }),
      ],
      useFactory: (config: ConfigService) => ({
        clients: {
          type: 'single',
          host: config.get('REDIS_HOST', 'localhost'),
          port: config.get('REDIS_PORT', 6379),
        },
      }),
    }),
  ],
})
export class AppModule {}

Here, three parameters set the policy: trip the breaker after failureThreshold failures within a windowMs rolling window, and stay open for openDurationMs before allowing probe traffic.

The Decorator

import { Injectable } from '@nestjs/common';
import { WithCircuitBreaker } from '@nestjs-redisx/circuit-breaker';

@Injectable()
export class PaymentsService {
  @WithCircuitBreaker({
    key: 'stripe',
    fallback: () => ({ queued: true }),
  })
  async charge(dto: ChargeDto) {
    return this.stripe.charge(dto);
  }
}

When the circuit is open, charge() doesn't even try to hit Stripe. It immediately returns the fallback, queuing the payment for a retry rather than outright failing the user's request.

The decorator is proxy-based, meaning it works on any @Injectable() method without needing a specific base class or manual service injection. Keys can be interpolated from method arguments, letting you scope a breaker per tenant, region, or partner:

@Injectable()
export class SearchService {
  @WithCircuitBreaker({ key: 'search:{0}', onOpen: 'skip' })
  async queryRegion(region: string, term: string) {
    return this.elastic.search(region, term);
  }
}

Each region has its own circuit. So, a failing eu-west cluster doesn't stop us-east queries. The onOpen: 'skip' option returns undefined instead of throwing — good for optional calls where a missing result is fine, but an error isn't.

The Programmatic API

For those cases where a decorator just won't cut it — dynamic keys, conditional wrapping, or code outside the DI graph — CircuitBreakerService provides the same tools directly:

import { Injectable, Inject } from '@nestjs/common';
import {
  CIRCUIT_BREAKER_SERVICE,
  type ICircuitBreakerService,
} from '@nestjs-redisx/circuit-breaker';

@Injectable()
export class WebhookDispatcher {
  constructor(
    @Inject(CIRCUIT_BREAKER_SERVICE)
    private readonly breaker: ICircuitBreakerService,
  ) {}

  async dispatch(endpoint: string, payload: unknown) {
    return this.breaker.execute(
      `webhook:${endpoint}`,
      () => this.http.post(endpoint, payload),
      { fallback: () => this.deadLetter.push(endpoint, payload) },
    );
  }
}

Besides execute(), the service offers recordSuccess and recordFailure for manual accounting. Decide what counts as a failure — maybe a 503 should trip the breaker, but a 404 shouldn't.

The State Machine

The breaker has three classic states:

Closed is normal. Calls go through, failures pile up in a rolling window. Hit the count within windowMs, and the breaker trips.

Open fails fast. Calls return the fallback (or throw, or skip) without hitting the dependency. This protects your service and the struggling dependency — no retry storm, no held connections. After openDurationMs, the breaker moves to half-open.

Half-open tests the waters. A limited number of calls go through to see if the dependency is back on its feet. Success closes the circuit; failure reopens it for another full cooldown. The cap is across all instances, so a recovering service gets just a few probes.

The Zombie Probe Problem

Half-open probes can create a subtle failure mode. A probe slot is claimed when a call is allowed through and released when the outcome is recorded. But if the process crashes mid-call? The slot is never released. With a small probe cap, the circuit might stay half-open indefinitely.

The plugin handles this with probeTimeoutMs: a probe that doesn't get its outcome recorded loses its slot after a timeout, freeing it for another instance.

When the State Store Itself Fails

A distributed breaker depends on something too. If Redis is unreachable, what should happen to the calls the breaker should protect?

There are two defensible answers, and the plugin lets you choose. Fail-open allows calls when the breaker state is unavailable — you lose protection but keep availability. Fail-closed rejects calls — you lose availability but ensure the breaker's protection is never absent.

Testing the Policy

Circuit breaker logic is tricky to test. It's time-based, so tests need fake timers. It's stateful, so tests need isolation. It's distributed, so tests want a Redis instance. The result? Often a slow, flaky suite.

This plugin separates policy from I/O. CircuitBreakerState is a pure state machine: no network calls, no Date.now(). Every time-based method takes an explicit now as an argument, making it fully deterministic and testable:

import { CircuitBreakerState } from '@nestjs-redisx/circuit-breaker';

it('trips after threshold failures inside the window', () => {
  const state = new CircuitBreakerState({
    failureThreshold: 3,
    windowMs: 10_000,
    openDurationMs: 30_000,
  });

  const t0 = 1_000_000;
  state.recordFailure(t0);
  state.recordFailure(t0 + 1_000);
  state.recordFailure(t0 + 2_000);

  expect(state.getStatus(t0 + 2_000)).toBe('open');
  expect(state.getStatus(t0 + 40_000)).toBe('half-open');
});

No fake timers, no waiting, no flakiness. Time is just numbers here.

For integration — decorator behavior, Redis coordination, Lua script correctness — the plugin uses @nestjs-redisx/testing, an in-memory driver. You can test the real code path without a Redis process:

import { Test } from '@nestjs/testing';
import { RedisTestingModule } from '@nestjs-redisx/testing';
import {
  CircuitBreakerPlugin,
  CIRCUIT_BREAKER_SERVICE,
  type ICircuitBreakerService,
} from '@nestjs-redisx/circuit-breaker';

it('fails fast once the circuit is open', async () => {
  const module = await Test.createTestingModule({
    imports: [
      RedisTestingModule.forRoot({
        plugins: [
          new CircuitBreakerPlugin({
            failureThreshold: 2,
            windowMs: 10_000,
            openDurationMs: 30_000,
          }),
        ],
      }),
    ],
  }).compile();
  await module.init();

  const breaker = module.get<ICircuitBreakerService>(CIRCUIT_BREAKER_SERVICE);
  const failing = async () => {
    throw new Error('upstream down');
  };

  await expect(breaker.execute('svc', failing)).rejects.toThrow();
  await expect(breaker.execute('svc', failing)).rejects.toThrow();

  let called = false;
  const result = await breaker.execute(
    'svc',
    async () => ((called = true), 'live'),
    { fallback: () => 'fallback' },
  );

  expect(result).toBe('fallback');
  expect(called).toBe(false);
});

And there you have it. @nestjs-redisx/circuit-breaker shares state through Redis, with atomic Lua transitions, providing a decorator and a programmatic API. It handles those tricky operational edge cases and keeps the policy layer pure for easy testing.

For a deep dive, check out the full reference or the GitHub repo.