Circuit breaker, retry and timeout in Node.js microservices — and why you need all three

A microservice architecture has one failure mode that a monolith never had: a dependency that is almost working. Not down — down is easy, the connection is refused and you know in a millisecond. Almost working means it answers one call in three, or it answers everything but takes four seconds to do it. await has no opinion about that. It waits. Your request handler waits with it, the HTTP client in front of you waits, and after enough of these pile up, the service that was perfectly healthy a minute ago is now also "almost working".

The standard answer is three guards that every mature RPC stack has some version of: a timeout (stop waiting), a retry policy (try again when it makes sense) and a circuit breaker (stop trying when it doesn't). In Node.js you usually get these from a library you wrap around each call — cockatiel, opossum, or p-retry plus something home-grown. In Moleculer they are broker options: one config block, applied to every service-to-service call in the cluster, with per-action overrides when one endpoint needs different rules.

This post switches each of them on, one at a time, against a dependency that we break on purpose, and shows the actual output. Every snippet below was run; the logs are pasted, not typed. The runnable code is in the example repo (run.sh reproduces every demo).

If you are new to Moleculer: a service is a plain object with actions (callable functions), and a broker runs services and routes calls between nodes over a transporter (NATS here). broker.call("inventory.check", params) is a request/response RPC that may land in-process or on another machine. The service discovery post covers how the broker knows where inventory.check lives.

The setup: a dependency with knobs

Three services. inventory is the one we abuse — it has runtime knobs so a single instance can be made slow or flaky while its siblings stay healthy:

// services/inventory.service.js
const { Errors } = require("moleculer");

module.exports = {
  name: "inventory",

  created() {
    this.knobs = {
      failRate: Number(process.env.FAIL_RATE || 0),   // 0..1 — share of calls that throw
      latency: Number(process.env.LATENCY_MS || 0),   // ms — artificial slowness
      failEvery: Number(process.env.FAIL_EVERY || 0), // 2 = every 2nd call fails (deterministic)
    };
    this.calls = 0;
  },

  actions: {
    check: {
      params: { sku: "string" },
      async handler(ctx) {
        this.calls++;
        const { failRate, latency, failEvery } = this.knobs;
        if (latency) await new Promise((r) => setTimeout(r, latency));
        const fail = (failEvery && this.calls % failEvery === 0) || Math.random() < failRate;
        if (fail) {
          // 503 + retryable: "try again, it might work"
          throw new Errors.MoleculerRetryableError("inventory database unavailable", 503, "DB_UNAVAILABLE");
        }
        return { sku: ctx.params.sku, inStock: 42, servedBy: this.broker.nodeID };
      },
    },

    configure: {
      params: { failRate: { type: "number", optional: true }, latency: { type: "number", optional: true }, failEvery: { type: "number", optional: true } },
      handler(ctx) {
        Object.assign(this.knobs, ctx.params);
        return this.knobs;
      },
    },
  },
};

orders.create calls inventory.check and then shipping.quote in sequence (we need a two-hop chain for the timeout section), and shipping.quote can be slowed down with SHIPPING_LATENCY_MS. Nothing else in them is interesting.

The config is shared by every node, and the three guards are switched on with environment variables so that each demo can show the same code with the feature off and then on:

// moleculer.config.js
module.exports = {
  nodeID: process.env.NODE_NAME || `node-${process.pid}`,
  transporter: process.env.TRANSPORTER || "nats://localhost:4222",

  requestTimeout: Number(process.env.REQUEST_TIMEOUT_MS || 0), // 0 = no timeout (the default!)

  retryPolicy: {
    enabled: process.env.RETRY === "1",
    retries: 3,
    delay: 100,      // first wait; then ×factor, capped at maxDelay
    maxDelay: 1000,
    factor: 2,
    check: (err) => err && !!err.retryable, // only errors that say "try again" — never a 4xx
  },

  circuitBreaker: {
    enabled: process.env.CB === "1",
    threshold: 0.5,        // open when ≥ 50 % of the window failed…
    minRequestCount: 5,    // …but only after at least 5 calls (default 20)
    windowTime: 60,        // seconds the counters are kept
    halfOpenTime: 5 * 1000, // ms until a single probe call is let through (default 10 s)
    check: (err) => err && err.code >= 500, // what counts as a failure
  },

  logger: { type: "Console", options: { level: process.env.LOG_LEVEL || "warn", formatter: "short" } },
};

Two things in that block are worth saying out loud before we run anything. All three guards are off by default, and requestTimeout defaults to 0, which means no timeout at all. A fresh Moleculer project will wait forever for a dead node, exactly like a fresh fetch() will. The framework gives you the tools; it does not guess your SLOs for you.

Workers are started with the standard runner, one process per node, e.g. NODE_NAME=inventory-bad FAIL_RATE=1 SERVICES=inventory SERVICEDIR=services npx moleculer-runner --config moleculer.config.js. The client (client.js) is a broker with no services that joins the same NATS and calls inventory.check in a loop, printing which node answered and how long it took.

1. Retry: try again, but only when it makes sense

inventory-1 runs with FAIL_EVERY=2 — every second call throws the retryable 503. Ten calls, retry policy off:

--- no retry policy (the default)
  ✓ inventory-1 inStock=42 (7 ms)
  ✗ MoleculerRetryableError (503) @inventory-1: inventory database unavailable (5 ms)
  ✓ inventory-1 inStock=42 (2 ms)
  ✗ MoleculerRetryableError (503) @inventory-1: inventory database unavailable (2 ms)
  ✓ inventory-1 inStock=42 (4 ms)
  ✗ MoleculerRetryableError (503) @inventory-1: inventory database unavailable (4 ms)
  ✓ inventory-1 inStock=42 (3 ms)
  ✗ MoleculerRetryableError (503) @inventory-1: inventory database unavailable (4 ms)
  ✓ inventory-1 inStock=42 (4 ms)
  ✗ MoleculerRetryableError (503) @inventory-1: inventory database unavailable (4 ms)

Fifty percent error rate, as designed. Same workers, same client, RETRY=1:

--- RETRY=1 (retries: 3, delay: 100 ms, factor 2)
  ✓ inventory-1 inStock=42 (28 ms)
[21:02:08.093Z] WARN  BROKER: Retry to call 'inventory.check' action after 100 ms... { requestID: 'aeba50eb-…', attempts: 1 }
  ✓ inventory-1 inStock=42 (122 ms)
[21:02:08.220Z] WARN  BROKER: Retry to call 'inventory.check' action after 100 ms... { requestID: 'b4405bb3-…', attempts: 1 }
  ✓ inventory-1 inStock=42 (120 ms)
[21:02:08.337Z] WARN  BROKER: Retry to call 'inventory.check' action after 100 ms... { requestID: '533175a5-…', attempts: 1 }
  ✓ inventory-1 inStock=42 (116 ms)
  … (same for the remaining six)

Ten out of ten succeed. Nine of them were retried once (the first call happened to hit an odd count, after that every call's first attempt lands on an even one), each paying the 100 ms delay — visible in the timings. Nothing in the service or the client changed; the retry lives in the broker of the caller, which is the only place it can live, because the callee cannot retry itself.

Three details that matter more than the happy path:

Which errors get retried is a property of the error, not of the policy. The default check is err => err.retryable. MoleculerRetryableError (and MoleculerServerError, which extends it) set that flag; the plain MoleculerError and every validation/4xx error do not. So a ValidationError is never retried no matter what you configure, and a 503 from a database that might come back is. When you throw from your own code, you decide — new Errors.MoleculerRetryableError(msg, 503, "DB_UNAVAILABLE") versus new Errors.MoleculerError(msg, 409, "DUPLICATE"). Think of it as the callee telling the caller whether trying again could possibly help.

The backoff is exponential and capped: delay, then delay × factor, then × factor again, never above maxDelay. With the values above that is 100 → 200 → 400 ms for the three retries, so a call that never succeeds costs about 700 ms of waiting on top of the attempts themselves. Set maxDelay with that arithmetic in mind.

Retries are per call, and they change which node answers. The retry goes through endpoint selection again, so with two instances of inventory a retry after a failure on one of them will typically land on the other. That is the failover behaviour you want — and the reason retries only make sense on idempotent actions. inventory.check can be run twice with no harm; payments.charge cannot. For those, either leave retries off for that action or make the action itself idempotent with a request key. Per-action and per-call overrides exist for exactly this:

// per action — in the service schema
actions: {
  charge: {
    retryPolicy: { enabled: false },
    async handler(ctx) { /* … */ },
  },
}

// per call — in the calling code
await ctx.call("inventory.check", { sku }, { retries: 5 });

2. Circuit breaker: stop asking a node that keeps saying no

Two inventory nodes now: inventory-good and inventory-bad (FAIL_RATE=1, fails everything). Round-robin sends every second call to the broken one, and without a breaker nothing ever changes that:

--- no circuit breaker: the broken node keeps getting every 2nd call
  ✗ # 1 MoleculerRetryableError (503) @inventory-bad: inventory database unavailable (14 ms)
  ✓ # 2 inventory-good inStock=42 (6 ms)
  ✗ # 3 MoleculerRetryableError (503) @inventory-bad: inventory database unavailable (6 ms)
  ✓ # 4 inventory-good inStock=42 (7 ms)
  ✗ # 5 MoleculerRetryableError (503) @inventory-bad: inventory database unavailable (10 ms)
  ✓ # 6 inventory-good inStock=42 (37 ms)
  … (alternating like this for all 12)

With CB=1 on the client. The client also registers listeners for the three $circuit-breaker.* events the broker broadcasts, printed as coloured dots. After the 12th call the script "repairs" the bad node with a targeted call (broker.call("inventory.configure", { failRate: 0 }, { nodeID: "inventory-bad" }) — nodeID in the call options bypasses load balancing) and keeps calling every 600 ms:

--- CB=1: the breaker isolates inventory-bad, then we fix it and it comes back
  ✓ # 1 inventory-good inStock=42 (11 ms)
  ✗ # 2 MoleculerRetryableError (503) @inventory-bad: inventory database unavailable (29 ms)
  ✓ # 3 inventory-good inStock=42 (5 ms)
  ✗ # 4 MoleculerRetryableError (503) @inventory-bad: inventory database unavailable (12 ms)
  ✓ # 5 inventory-good inStock=42 (6 ms)
  ✗ # 6 MoleculerRetryableError (503) @inventory-bad: inventory database unavailable (6 ms)
  ✓ # 7 inventory-good inStock=42 (6 ms)
  ✗ # 8 MoleculerRetryableError (503) @inventory-bad: inventory database unavailable (5 ms)
  ✓ # 9 inventory-good inStock=42 (9 ms)
  🔴 circuit OPENED   for inventory.check on inventory-bad (5/5 failed)
  ✗ #10 MoleculerRetryableError (503) @inventory-bad: inventory database unavailable (5 ms)
  ✓ #11 inventory-good inStock=42 (7 ms)
  ✓ #12 inventory-good inStock=42 (6 ms)
  → inventory.configure on inventory-bad: failRate=0 (the node is "fixed")
  ✓ #13 inventory-good inStock=42 (3 ms)
  ✓ #14 inventory-good inStock=42 (4 ms)
  ✓ #15 inventory-good inStock=42 (5 ms)
  ✓ #16 inventory-good inStock=42 (5 ms)
  ✓ #17 inventory-good inStock=42 (4 ms)
  ✓ #18 inventory-good inStock=42 (4 ms)
  ✓ #19 inventory-good inStock=42 (5 ms)
  ✓ #20 inventory-good inStock=42 (4 ms)
  🟡 circuit HALF-OPEN for inventory.check on inventory-bad — one probe call allowed
  🟢 circuit CLOSED   for inventory.check on inventory-bad — back in rotation
  ✓ #21 inventory-bad inStock=42 (6 ms)
  ✓ #22 inventory-good inStock=42 (17 ms)
  ✓ #23 inventory-bad inStock=42 (12 ms)
  ✓ #24 inventory-good inStock=42 (4 ms)

Reading it top to bottom:

• Calls 2–10 fail on inventory-bad. The fifth failure (call #10) satisfies minRequestCount: 5 with a 100 % failure rate, above the threshold: 0.5, so the breaker opens — the event fires while call #10 is still returning its error.
• From #11 on, inventory-bad gets nothing. Not because the client checks a flag before each call: the breaker sets the endpoint's state to unavailable in the local registry, so the load balancer simply does not see it any more. Every call goes to inventory-good. The error rate drops from 50 % to zero without touching the broken node.
• We fix the node at #12. Nothing changes immediately — the breaker does not know, and that is correct. Five seconds after it opened (halfOpenTime) it goes half-open: the endpoint is made visible again for exactly one call, and hidden again until that probe completes.
• The probe (#21) succeeds, the breaker closes, and round-robin resumes alternating. Had the probe failed, the breaker would have gone straight back to open for another 5 s.

Two properties that are easy to get wrong when you build this yourself, and that the broker gets right because it is the load balancer:

The breaker is per endpoint — per (node, action) pair — not per service. inventory.check on inventory-bad was open while inventory.check on inventory-good was closed. A breaker per service name would have taken the healthy node down with the broken one. A breaker per node would have blocked inventory.configure too, and we could not have repaired it through the same channel.

Failure counting happens on the caller, and the caller decides what a failure is. The default check is err.code >= 500: server errors and timeouts count, validation errors and not-founds do not. That is per caller; a caller with different needs (a batch job that treats slow responses as failures) can register its own check. There is also a wrinkle in the source worth knowing: an error that originates on a third node (an inventory handler that itself called something that failed) is not counted against the inventory endpoint — only errors raised by the node you called, or locally, are.

Per-action overrides use the same shape as the broker option:

actions: {
  check: {
    circuitBreaker: { threshold: 0.3, minRequestCount: 10 },
    async handler(ctx) { /* … */ },
  },
}

And the $circuit-breaker.opened / half-opened / closed events are ordinary broker events — the payload has nodeID, action, failures, count, rate — so a monitoring service can subscribe to them like to anything else. If you have metrics enabled there are gauges for active open and half-open breakers as well.

3. Timeout: the one that is off by default

inventory-slow answers after 3 seconds. The client calls with a per-call timeout of 1 s:

--- only the slow node exists
[21:04:56.031Z] WARN  BROKER: Request 'inventory.check' is timed out. { requestID: 'be7093ec-…', nodeID: 'inventory-slow', timeout: 1000 }
  ✗ #1 timeout=1000 RequestTimeoutError (504) @inventory-slow: Request is timed out when call 'inventory.check' action on 'inventory-slow' node. (1007 ms)
  ✗ #2 timeout=1000 RequestTimeoutError (504) @inventory-slow: … (1011 ms)
  ✗ #3 timeout=1000 RequestTimeoutError (504) @inventory-slow: … (1005 ms)
  ✗ #4 timeout=1000 RequestTimeoutError (504) @inventory-slow: … (1003 ms)

That is all a timeout does on its own: it turns a 3-second wait into a 1-second RequestTimeoutError. Useful — your request handler is unblocked and can respond with a 504 — but the user still got nothing. Now look at the error's two properties: code 504 and retryable: true. That is where it composes with the previous two sections.

Start a healthy inventory-fast next to the slow one, and enable retry:

--- a healthy node joins; RETRY=1 turns the timeout into a failover
[21:05:04.302Z] WARN  BROKER: Request 'inventory.check' is timed out. { requestID: '50aa1ded-…', nodeID: 'inventory-slow', timeout: 1000 }
[21:05:04.305Z] WARN  BROKER: Retry to call 'inventory.check' action after 100 ms... { requestID: '50aa1ded-…', attempts: 1 }
  ✓ #1 timeout=1000 inventory-fast inStock=42 (1164 ms)
[21:05:05.451Z] WARN  BROKER: Request 'inventory.check' is timed out. { requestID: '7d3cfb02-…', nodeID: 'inventory-slow', timeout: 1000 }
[21:05:05.452Z] WARN  BROKER: Retry to call 'inventory.check' action after 100 ms... { requestID: '7d3cfb02-…', attempts: 1 }
  ✓ #2 timeout=1000 inventory-fast inStock=42 (1132 ms)
  … (#3, #4 identical)

Every call now succeeds — the timeout is retryable, the retry re-selects an endpoint, round-robin hands it to inventory-fast. But every call also costs 1.1 s, because round-robin keeps starting on the slow node. Retry is the wrong tool for a node that is consistently slow; it fixes the individual call and keeps paying the tax. Add the breaker:

--- RETRY=1 CB=1: timeouts count as failures, so the slow node gets cut off
  ✓ #1 timeout=1000 inventory-fast inStock=42 (11 ms)
  ✓ #2 timeout=1000 inventory-fast inStock=42 (1110 ms)
  ✓ #3 timeout=1000 inventory-fast inStock=42 (1108 ms)
  ✓ #4 timeout=1000 inventory-fast inStock=42 (1109 ms)
  ✓ #5 timeout=1000 inventory-fast inStock=42 (1109 ms)
  🔴 circuit OPENED   for inventory.check on inventory-slow (5/5 failed)
  ✓ #6 timeout=1000 inventory-fast inStock=42 (1108 ms)
  ✓ #7 timeout=1000 inventory-fast inStock=42 (6 ms)
  ✓ #8 timeout=1000 inventory-fast inStock=42 (7 ms)
  ✓ #9 timeout=1000 inventory-fast inStock=42 (7 ms)
  ✓ #10 timeout=1000 inventory-fast inStock=42 (5 ms)

Five timeouts (504 ≥ 500, so they count) open the breaker on inventory-slow, and from #7 on the calls are 5 ms. Nobody wrote any failover logic; three independent guards, each doing its narrow job, produced it. This composition is the actual point of the post — and it is also why the broker applies them in a fixed order, which matters:

• Fallback is the outermost layer — it only kicks in after everything else has given up.
• Retry wraps Timeout, so each attempt gets the full timeout; a retried call can take retries × timeout in the worst case. Budget accordingly.
• Timeout wraps the Circuit breaker, so the caller's timeout fires outside the breaker. A slow endpoint is counted only when the callee's own timeout error (504) comes back through the breaker, and that happens only while retry keeps the request alive: with retries off, the caller drops the pending request first, the 504 arrives as an orphan response, and the breaker never sees it. Retry is on in this section, which is why the breaker opened above.
• Bulkhead (section 6) is innermost and runs on the callee.

The timeout can be set on three levels, most specific wins: per call ({ timeout: 1000 } in the options, as above), per action (timeout: 1000 in the action schema — the callee saying "I should never take longer than this"), and per broker (requestTimeout). Set the broker-level one in every production config. A wrong value is better than 0.

4. Distributed timeout: the budget travels with the request

This is the part that a per-call timeout library cannot do for you. orders.create calls inventory.check (800 ms) and then shipping.quote (800 ms). The client calls orders.create with a timeout. First with a 3 s budget:

--- 3 s budget: fits
  ✓ orders.create timeout=3000 → inStock=42 timings={"inventory":813,"shipping":817} (1733 ms)

Then with 1.5 s, which is less than the two hops need:

--- 1.5 s budget: the second hop only gets what is left
[21:05:23.513Z] WARN  BROKER: Request 'orders.create' is timed out. { requestID: '6cd10b50-…', nodeID: 'app-1', timeout: 1500 }
  ✗ orders.create timeout=1500 → RequestTimeoutError (504) @app-1: Request is timed out when call 'orders.create' action on 'app-1' node. (1506 ms)
--- what app-1 (the orders node) logged meanwhile:
[21:05:23.513Z] WARN  BROKER: Request 'shipping.quote' is timed out. { requestID: '6cd10b50-…', nodeID: 'app-1', timeout: 694.221156 }

Look at the second log line, from the orders node. The shipping.quote call was made with a timeout of 694.22 ms — nobody wrote that number. The client's 1500 ms was carried in the request to app-1; app-1 spent ~806 ms on the inventory hop; when it made the next call, the broker subtracted the elapsed time and passed the remainder down. Both sides timed out at the same millisecond (23.513Z), with the same requestID, and the client got its answer at 1506 ms, not at 1600 or 2300.

In distributed-systems vocabulary this is deadline propagation — the thing gRPC does with deadlines and that most HTTP stacks do not do at all. Without it, a 1.5 s timeout at the edge and a 1.5 s timeout on each inner hop means a request can keep a chain of services busy for 3 s after the caller has already given up on it. With it, the whole tree is cut at the same instant. If the budget is already exhausted when a call is about to be made, the broker does not even send it — you get a RequestSkippedError instead, and that one is deliberately not retryable: there is no budget left to retry with.

5. Fallback: degrade instead of failing

Retry and breaker decide whether to keep trying. When they give up, fallbackResponse decides what the caller gets instead of an exception. Inventory is fully down (FAIL_RATE=1, single node):

const res = await broker.call("inventory.check", { sku: "SKU-1" }, {
  fallbackResponse: (ctx, err) => {
    console.log(`  ↩ fallback used because: ${err.name} (${err.code})`);
    return { sku: ctx.params.sku, inStock: null, stale: true };
  },
});

[21:05:27.838Z] WARN  BROKER: The 'inventory.check' request is failed. Return fallback response. { requestID: '06fa6dc8-…', err: 'inventory database unavailable' }
  ↩ fallback used because: MoleculerRetryableError (503) @inventory-1
  result: { sku: 'SKU-1', inStock: null, stale: true }

fallbackResponse can be a value or a function of (ctx, err) — the function form is where you read from a cache, return the last known value, or return a "we'll confirm by email" answer. The broker sets ctx.fallbackResult = true so tracing and metrics can tell a real answer from a degraded one. On the callee side there is a mirror of this: an action can declare fallback: "methodName" (or a function) in its schema, which runs when the handler throws — useful for the "serve from cache when the DB is down" case where the service knows its own fallback better than every caller does.

A product page that shows "stock: unknown" is a product page. A product page that shows a 500 is not. The fallback is where the difference is made, and it is one option on the call.

6. Bulkhead: don't let one endpoint sink the ship

The last guard is on the callee. orders.report is expensive (300 ms of pretend work) and declares a bulkhead of two concurrent executions and a queue of three:

report: {
  bulkhead: { enabled: true, concurrency: 2, maxQueueSize: 3 },
  async handler(ctx) {
    await new Promise((r) => setTimeout(r, 300));
    return { ok: true, servedBy: this.broker.nodeID };
  },
},

Ten calls fired at once with Promise.allSettled:

  ✓ #0 ok
  ✓ #1 ok
  ✓ #2 ok
  ✓ #3 ok
  ✓ #4 ok
  ✗ #5 QueueIsFullError (429) @app-1: Queue is full. Request 'orders.report' action on 'client-3475145' node is rejected.
  ✗ #6 QueueIsFullError (429) @app-1: Queue is full. …
  ✗ #7 QueueIsFullError (429) @app-1: Queue is full. …
  ✗ #8 QueueIsFullError (429) @app-1: Queue is full. …
  ✗ #9 QueueIsFullError (429) @app-1: Queue is full. …
  total 951 ms

Two run, three wait, five are rejected immediately with a 429 — they do not sit in an unbounded queue eating memory and dragging the p99 of everything else on that node. The five accepted ones finish in three batches (2 + 2 + 1 ≈ 950 ms). The name comes from the watertight compartments in a ship's hull: a flood in one does not sink the whole thing. The bulkhead can be declared per action, as here, or for the whole broker (bulkhead: { enabled: true, concurrency: 10, maxQueueSize: 100 }). Note that QueueIsFullError is marked retryable: with a retry policy on the caller, a rejected call is retried after the backoff, and with several instances of the service that usually lands on a less busy one — a reasonable default. With a single instance it means retrying against the same full queue, which is one more place where a per-action retryPolicy: { enabled: false } earns its keep.

(One quirk visible in the log: the node named in the error message is the caller, because ctx.nodeID on the callee side refers to the sender of the request. The @app-1 prefix — which is err.nodeID, the node that threw — is the one that tells you where the queue actually lives.)

What to put in production

A reasonable baseline that I would start any Moleculer 0.15 project from:

module.exports = {
  requestTimeout: 5 * 1000,
  retryPolicy: { enabled: true, retries: 3, delay: 100, maxDelay: 2000, factor: 2 },
  circuitBreaker: { enabled: true, threshold: 0.5, minRequestCount: 20, windowTime: 60, halfOpenTime: 10 * 1000 },
  bulkhead: { enabled: true, concurrency: 50, maxQueueSize: 200 },
};

Then, per action, turn retries off on anything non-idempotent, set a tighter timeout on anything user-facing, and add fallbacks where a degraded answer is better than none. The defaults you are overriding — no timeout, no retry, no breaker — are the right defaults for a framework that does not know your system, and the wrong ones for any system that has users.

The framework's contribution is not that it has a circuit breaker; every resilience library has one. It is that the breaker, the retry, the timeout, the load balancer and the registry are the same component, so the timeout knows how much budget is left, the retry knows there is another node, and the breaker can take an endpoint out of rotation instead of throwing faster. Three guards, one config block, and the failover in section 3 falls out of it.

---

Code for this post: examples/06-resilience — run.sh reproduces all six demos against a local NATS. Moleculer 0.15.2, Node 22.

Previous in the series: Service discovery and load balancing in Node.js. Next: event-driven architecture — pub/sub with NATS, Kafka and Redis.

Comments

[shieldx]

Việc kết hợp cả ba cơ chế này thực sự là bài toán sống còn khi hệ thống bắt đầu scale. Mình từng gặp trường hợp chỉ dùng retry mà không có circuit breaker, kết quả là khi service downstream bị quá tải, các request retry dội ngược lại vô tình tạo ra một đợt DDoS tự thân, khiến hệ thống sập nhanh hơn. Một điểm quan trọng khác là cần phải tính toán timeout cực kỳ cẩn thận để nó không thấp hơn thời gian xử lý trung bình của service phụ thuộc, nếu không bạn sẽ vô tình ngắt các request hợp lệ. Việc phân tách rõ ràng giữa timeout budget và retry logic giúp tránh tình trạng lãng phí tài nguyên khi service đích đã thực sự "chết".