DEV Community: Icebob

Moleculer vs gRPC vs tRPC: an RPC protocol is not a service layer

Icebob — Tue, 20 Oct 2026 13:00:16 +0000

Ask an AI assistant "how should my Node.js services talk to each other" and you will get gRPC or tRPC. Both are good answers to a question — just not the same question you might be asking. gRPC and tRPC are ways to call a function on another machine. Moleculer is a service layer: it also calls functions on other machines, but it is mostly about the things around the call — finding the machine, choosing between three copies of it, giving up on a slow one, publishing "this happened" to whoever cares. Comparing them head-to-head is like comparing HTTP with Express. This post does it anyway, because the confusion is real, and because the honest answer is that they overlap in one place and differ everywhere else.

The plan: build the same tiny products service three times, then throw the same three problems at each — a second instance, a dead instance, a slow instance — and watch what each tool does out of the box. Then the parts where Moleculer is genuinely weaker (types, other languages, browsers), the throughput shape on one box, and two setups where you use them together, which is where most real systems end up.

Every snippet was executed and the output pasted. The code is in the example repo; run.sh reproduces everything below in about two minutes.

Versions: @grpc/grpc-js 1.14.4 + @grpc/proto-loader 0.8.1, @trpc/server + @trpc/client 11.18 with zod 4.5, Moleculer 0.15.2, Node 22. Moleculer terms, glossed once: a service is an object with actions (callable functions) and events (pub/sub handlers); a broker hosts services on a node (a process) and talks to other nodes over a transporter (NATS here, plain TCP later); the registry is the broker's live list of which node has which action — the thing gRPC calls a resolver and Kubernetes calls service discovery.

What each one is

gRPC is a protocol: HTTP/2 framing, Protocol Buffers for the payload, a .proto file as the contract, and generated (or, in Node, runtime-loaded) stubs in a dozen languages. It has some client-side features — deadlines, retries and load-balancing policies configured through a "service config" — but they are opt-in and they operate on a list of addresses someone else has to provide. Discovery, health, circuit breaking, pub/sub: not gRPC's job; that is what service meshes and xDS are for.

tRPC is a TypeScript library: you write a router of procedures with zod inputs, export its type, and a client in another TypeScript project gets full autocompletion and return types with no codegen and no schema file. On the wire it is plain HTTP + JSON (batched). It was designed for a full-stack TypeScript app — a Next.js frontend calling its own backend — and it is superb at that. It has no notion of multiple servers at all: a client has one URL.

Moleculer is a framework: services, actions, events, a registry, load balancing, timeouts, retries, circuit breaker, bulkhead, caching, tracing, metrics, and pluggable transport. The "RPC" part of it — broker.call("products.get", { id: 2 }) — is a thin JSON-over-transporter message that the previous seven posts of this series have used. It is Node.js only, and its TypeScript story is weaker than tRPC's (post 3 is the honest version).

That is the whole post in one picture. The rest is proof.

The same service, three times

Three products, get by id, list with a limit, and every response says which instance served it (servedBy), because that is what we will be watching.

gRPC — the contract lives in a .proto file:

syntax = "proto3";
package products;

service Products {
  rpc Get (GetRequest) returns (Product);
  rpc List (ListRequest) returns (ProductList);
}

message GetRequest  { int32 id = 1; }
message ListRequest { int32 limit = 1; }
message Product     { int32 id = 1; string name = 2; double price = 3; string served_by = 4; }
message ProductList { repeated Product items = 1; }

// grpc/server.js
const pkg = grpc.loadPackageDefinition(
  protoLoader.loadSync(path.join(__dirname, "products.proto"), { keepCase: true, longs: Number, defaults: true })
).products;

const server = new grpc.Server();
server.addService(pkg.Products.service, {
  Get(call, callback) {
    const product = PRODUCTS.find((p) => p.id === call.request.id);
    if (!product) return callback({ code: grpc.status.NOT_FOUND, message: `product ${call.request.id} not found` });
    callback(null, { ...product, served_by: NAME });
  },
  List(call, callback) {
    callback(null, { items: PRODUCTS.slice(0, call.request.limit || 10).map((p) => ({ ...p, served_by: NAME })) });
  },
});
server.bindAsync(`0.0.0.0:${PORT}`, grpc.ServerCredentials.createInsecure(), () => { /* … */ });

tRPC — the contract is the code:

// trpc/server.ts
const t = initTRPC.create();

export const appRouter = t.router({
  products: t.router({
    get: t.procedure
      .input(z.object({ id: z.number().int() }))
      .query(({ input }) => {
        const product = PRODUCTS.find((p) => p.id === input.id);
        if (!product) throw new TRPCError({ code: "NOT_FOUND", message: `product ${input.id} not found` });
        return { ...product, servedBy: NAME };
      }),
    list: t.procedure
      .input(z.object({ limit: z.number().int().default(10) }))
      .query(({ input }) => PRODUCTS.slice(0, input.limit).map((p) => ({ ...p, servedBy: NAME }))),
  }),
});

// The client imports this TYPE only — no code, no schema file, no codegen.
export type AppRouter = typeof appRouter;

createHTTPServer({ router: appRouter }).listen(PORT);

Moleculer — a service object; the parameter schema is a fastest-validator shorthand:

// moleculer/products.service.js
module.exports = {
  name: "products",
  actions: {
    get: {
      params: { id: "number|integer|convert" },
      async handler(ctx) {
        const product = PRODUCTS.find((p) => p.id === ctx.params.id);
        if (!product) throw new Errors.MoleculerClientError(`product ${ctx.params.id} not found`, 404, "NOT_FOUND", { id: ctx.params.id });
        return { ...product, servedBy: NAME };
      },
    },
    list: {
      params: { limit: { type: "number", integer: true, default: 10 } },
      handler(ctx) {
        return PRODUCTS.slice(0, ctx.params.limit).map((p) => ({ ...p, servedBy: NAME }));
      },
    },
  },
};

// moleculer/node.js — a process that hosts the service
const broker = new ServiceBroker(require("./moleculer.config.js")); // nodeID, transporter: "nats://localhost:4222", …
broker.createService(require("./products.service.js"));
broker.start();

And one call to each, hit and miss:

--- grpc: get 2, then get 99
  → {"id":2,"name":"Mouse","price":19.9,"served_by":"products-1"}
  → error code 5 NOT_FOUND: product 99 not found
--- trpc: get 2, then get 99
  → {"id":2,"name":"Mouse","price":19.9,"servedBy":"products-1"}
  → error NOT_FOUND (HTTP 404): product 99 not found
--- mol: get 2, then get 99
  → {"id":2,"name":"Mouse","price":19.9,"servedBy":"products-1"}
  → error MoleculerClientError (404 NOT_FOUND): product 99 not found

Same behaviour, three idioms. The client side is where they start to look different:

// gRPC: a stub from the proto, a target address, a callback API
const client = new pkg.Products("127.0.0.1:50051", grpc.credentials.createInsecure());
client.Get({ id: 2 }, (err, res) => …);

// tRPC: a typed proxy from the router type and one URL
const client = createTRPCClient<AppRouter>({ links: [httpBatchLink({ url: "http://127.0.0.1:4001" })] });
const p = await client.products.get.query({ id: 2 }); // p is { id: number; name: string; price: number; servedBy: string }

// Moleculer: a broker that joins the cluster; no address of anything
const broker = new ServiceBroker({ transporter: "nats://localhost:4222" });
await broker.start();
const p = await broker.call("products.get", { id: 2 }); // p is `any` — see "Type safety" below

At this point tRPC is the nicest and Moleculer is the heaviest (it needs NATS running). Now the interesting part.

Problem 1: a second instance

Start products-2 next to products-1 for all three and make four calls.

--- gRPC: ipv4:127.0.0.1:50051,127.0.0.1:50052 + round_robin service config
  call 1: served by products-1
  call 2: served by products-1
  call 3: served by products-2
  call 4: served by products-1
--- tRPC: one URL — the second server on :4002 is simply never called
  call 1: served by products-1
  call 2: served by products-1
  call 3: served by products-1
  call 4: served by products-1
--- Moleculer: nothing configured, the registry knows both nodes
  products.get endpoints: products-1, products-2
  call 1: served by products-2
  call 2: served by products-1
  call 3: served by products-2
  call 4: served by products-1

gRPC can balance, and it took this to make it do so:

// grpc/client.js — a resolver that yields several addresses, plus a service config
const serviceConfig = { loadBalancingConfig: [{ round_robin: {} }], methodConfig: [/* retry policy, later */] };
const client = new pkg.Products("ipv4:127.0.0.1:50051,127.0.0.1:50052", grpc.credentials.createInsecure(), {
  "grpc.service_config": JSON.stringify(serviceConfig),
});

The default policy is pick_first, and the addresses come from the target string — in production from DNS or xDS. Either way the client is told the list; nobody tells it when the list changes. (The first two calls both landing on products-1 is grpc-js starting to route before the second connection was ready — round robin over connected subchannels.)

tRPC has one URL. If you want two servers, you put a reverse proxy in front and it is the proxy that balances; tRPC does not know or care. That is not a flaw — it is the shape of the tool — but it means "tRPC between microservices" quietly implies "plus nginx/Envoy/an ingress and a plan for how they find the instances".

Moleculer's client — which is just another node — learned about both instances from the registry the moment it started, and round-robins between them (the discovery post covers the strategies). No address was written anywhere.

Problem 2: an instance dies

Stop products-2 and call again.

--- gRPC: the closed connection drops out of round_robin; the static list still names it
  call 1: served by products-1
  call 2: served by products-1
  call 3: served by products-1
  call 4: served by products-1
--- Moleculer: the node left the registry
  products.get endpoints: products-1
  call 1: served by products-1
  call 2: served by products-1
  call 3: served by products-1
  call 4: served by products-1

Both cope. gRPC noticed the TCP connection close and stopped picking that subchannel; it will keep trying to reconnect to :50052 in the background because the address is still on its list. Moleculer got a disconnect message from the node (a graceful stop) — for a crash it would take the heartbeat timeout, 6 s in this config — and removed it from the registry. tRPC is not in this section because one URL means there is nothing to fail over to.

Problem 3: an instance is slow

products-2 comes back, but every answer takes 800 ms. This is the case that separates "can call a remote function" from "can run a system", because a slow dependency is worse than a dead one: it ties up the caller.

gRPC has deadlines, per call:

client.Get({ id: 1 }, { deadline: Date.now() + 300 }, callback);

--- gRPC: deadline 300 ms against the slow server
  → DEADLINE_EXCEEDED after 306 ms — Deadline exceeded after 0.297s,name resolution: 0.028s,metadata filters: 0.003s,time to current attempt start: 0.001s,LB pick: 0.019s,remote_addr=127.0.0.1:50052

The deadline is propagated to the server too (the handler can check call.getDeadline()), which is a genuinely good design. But it is the only thing that happens. The round-robin client keeps sending every other call to the slow instance, and the retry policy — which we configured with retryableStatusCodes: ["UNAVAILABLE"], the only sensible default — does not fire on DEADLINE_EXCEEDED, because retrying a call that took too long, with the same deadline, is usually wrong:

--- gRPC: round robin keeps sending every second call to the slow instance
  call 1: served by products-2
  call 2: served by products-2
  call 3: served by products-1
  call 4: served by products-2

No latency awareness, no outlier ejection, no circuit breaker. Those exist in the gRPC ecosystem — in Envoy, in xDS-driven load balancing, in a mesh — but not in the client library.

tRPC has neither deadlines nor retries at the protocol level; you pass an AbortSignal (signal: AbortSignal.timeout(300)) and the server keeps working on the request. Retrying is the client application's problem, and with one URL a retry goes to the same server. Fine for a browser talking to a backend behind a load balancer; not a service-to-service story.

Moleculer treats the slow instance as a system problem, and the config is three blocks that the resilience post explains in detail:

// moleculer/moleculer.config.js
requestTimeout: 300, // a broker-wide default; per-call override: broker.call(…, { timeout })
retryPolicy: {
  enabled: true, // the policy must be enabled here for a per-call `retries` to have any effect
  retries: 0,    // …but by default we don't retry; the demo passes { retries: 1 } per call
  delay: 100, maxDelay: 1000, factor: 2,
  check: (err) => err && !!err.retryable, // timeouts and "service unavailable" are; a 404 is not
},
circuitBreaker: {
  enabled: true,
  threshold: 0.5,     // open when ≥ 50 % of the calls in the window failed…
  minRequestCount: 3, // …after at least 3 calls (default 20)
  windowTime: 60,
  halfOpenTime: 5000, // ms until a probe call is let through again
  check: (err) => err && err.code >= 500,
},

--- plain calls, 300 ms requestTimeout, one instance is slow:
  call 1: RequestTimeoutError after 307 ms — Request is timed out when call 'products.get' action on 'products-2' node. (node products-2)
  call 2: served by products-1 in 8 ms
  call 3: RequestTimeoutError after 302 ms — Request is timed out when call 'products.get' action on 'products-2' node. (node products-2)
  call 4: served by products-1 in 6 ms
--- same, with retries: 1 → the retry goes to another instance, and the breaker counts the timeouts:
  call 1: served by products-1 in 409 ms
  call 2: served by products-1 in 406 ms
  🔴 circuit OPENED for products.get on products-2 (3/3 failed)
  call 3: served by products-1 in 410 ms
  call 4: served by products-1 in 5 ms
--- the circuit is open for the slow endpoint; plain calls again:
  call 1: served by products-1 in 6 ms
  call 2: served by products-1 in 3 ms
  call 3: served by products-1 in 4 ms
  call 4: served by products-1 in 5 ms
  endpoint state: products-2=false, products-1=true

Three stages. Plain calls: the timeout works like a gRPC deadline (and, like gRPC's, it is propagated — the callee stops too). With retries: 1: the retry is routed by the registry to the other instance, so the caller sees a slow success instead of an error, and each timeout is counted against products-2. After three, the breaker opens that one endpoint — the node stays in the registry, it just receives no calls until a probe succeeds 5 s later — and call 4 onwards is back to 5 ms. The 🔴 line is the $circuit-breaker.opened event, which any service in the cluster can subscribe to; that is how you get an alert instead of a graph that looks odd.

One honest footnote, because it cost me an hour: a plain timeout with no retry does not trip the breaker. The breaker counts failures that come back from the callee; a caller-side timeout is raised outside it, so the callee's own "timed out" reply arrives to a request that has already been abandoned. With the retry policy enabled, the retry keeps the request alive long enough for that reply to be counted. The practical rule: if you enable circuitBreaker, enable retryPolicy too (even with retries: 0 as the default, the mechanism is then in place for the calls that ask for it). The resilience post's RETRY=1 CB=1 demo worked for exactly this reason, which I did not know when I wrote it.

Problem 4: a third instance appears

A deploy scaled the service up. products-3 starts on a port the clients have never heard of.

--- gRPC: the client's address list is static — :50053 is never used
  call 1: served by products-1
  call 2: served by products-1
  call 3: served by products-1
  call 4: served by products-1
--- Moleculer: a new node in the registry, no client change
  products.get endpoints: products-1, products-3
  call 1: served by products-3
  call 2: served by products-1
  call 3: served by products-3
  call 4: served by products-1

This is the difference in one screen. gRPC did nothing wrong; it was never told. The thing that would tell it — a DNS record that updates, an xDS control plane, Kubernetes — is the infrastructure Moleculer ships in the box. Whether you want that in the box or in the platform is a legitimate architectural choice, and the Kubernetes post is about exactly that; but it is a choice you have to make, and "just use gRPC" hides it.

Type safety, honestly

This is where the comparison flips.

tRPC is the best-in-class answer. The client's p.servedBy above is typed because the client was created from typeof appRouter. Rename a field on the server and every caller fails to compile. Zero build step, zero schema, one language. If both ends are TypeScript and both ends are yours, nothing else comes close.

gRPC gets there with codegen — protoc with a TypeScript plugin, or proto-loader-gen-types — and the .proto file is a contract that other languages honour too. It is more ceremony than tRPC and less pleasant (callbacks, keepCase decisions, int64 as strings), but it is the only one of the three whose types survive a language boundary.

Moleculer has the weakest story of the three, and there is no point pretending otherwise. broker.call returns Promise<any> unless you pass generics yourself — broker.call<Product, { id: number }>("products.get", { id }) — and nothing checks that "products.get" exists or that the generics match what the handler returns. Parameter validation is real but runtime (fastest-validator, or zod via a custom validator), so a wrong call fails at request time with a 422, not at compile time. The community moleculer-decorators package and the patterns in post 3 make services pleasant to write in TypeScript; they do not make calls between services type-checked. If end-to-end static typing across services is the thing you value most, tRPC wins, and the last section shows how to have both.

Other languages, browsers, streaming

gRPC is the polyglot option, full stop. A Go service and a Node service share the .proto and nothing else. It also has real streaming — server, client and bidirectional — over HTTP/2, which neither of the others matches (tRPC has subscriptions over SSE/WebSocket; Moleculer has streams for request/response and events for fan-out, not a bidirectional channel). What gRPC cannot do is talk to a browser without gRPC-Web and a proxy.

tRPC is the browser option. That is what it is for; httpBatchLink even collapses ten calls made in the same tick into one request. It does not exist outside TypeScript (there are OpenAPI bridges, with the type safety left at the door).

Moleculer is Node.js only — every node in the cluster is a Node process running the broker. Browsers reach it through moleculer-web, an API gateway service that maps HTTP routes to actions (the NestJS post shows one). Other languages reach it by speaking the transporter protocol — there are Go and Java ports of varying completeness — or, more sensibly, through a gRPC or HTTP boundary owned by one Node service, which is the second setup below.

Throughput shape

Same box, loopback, one server instance, one client process, 2000 calls: first one at a time, then all in flight. These are not benchmarks — the sequential numbers are dominated by the client library's per-call overhead on a small VM, and a real system is never one client on loopback. Read them as shapes.

  gRPC unary, sequential: 2000 calls in 8677 ms (230 req/s, 4.34 ms/call)
  gRPC unary, 2000 in flight: 1305 ms (1533 req/s)
  tRPC over HTTP, sequential: 2000 calls in 16500 ms (121 req/s, 8.25 ms/call)
  tRPC over HTTP, 2000 in flight (batched): 383 ms (5222 req/s)
  tRPC over HTTP, sequential: 2000 calls in 11217 ms (178 req/s, 5.61 ms/call)   ← httpLink, no batching
  tRPC over HTTP, 2000 in flight: 1905 ms (1050 req/s)
  Moleculer over nats, sequential: 2000 calls in 6473 ms (309 req/s, 3.24 ms/call)
  Moleculer over nats, 2000 in flight: 198 ms (10101 req/s)
  Moleculer over TCP, sequential: 2000 calls in 2121 ms (943 req/s, 1.06 ms/call)
  Moleculer over TCP, 2000 in flight: 226 ms (8850 req/s)

Three things worth taking from it. tRPC's batching is the reason its "in flight" number is thirteen times its sequential one: 2000 calls became 20 HTTP requests. Moleculer over NATS is the fastest of the three at concurrency because the transporter connection is one long-lived socket per node and the broker is fire-and-forget on send; over the TCP transporter (no NATS at all, nodes talk directly) it is also the fastest sequentially. And the per-call cost is a few milliseconds for all of them on this machine — the wire format is not where your latency budget goes; the slow instance in Problem 3 is.

Using them together

Most systems that ask "gRPC or Moleculer" end up with a service layer and an RPC protocol, because they solve different problems. Two setups, both running.

tRPC in front of a Moleculer cluster

The gateway is a Moleculer node with no services of its own, exposing a tRPC router whose procedures call the cluster. The browser gets tRPC's types; the calls get the registry, the balancer and the breaker.

// combined/gateway.ts
const broker = new ServiceBroker({ ...(config as BrokerOptions), nodeID: "trpc-gateway" });

type Product = { id: number; name: string; price: number; servedBy: string }; // broker.call is untyped — we declare

const toTRPCError = (err: any) =>
  err?.code === 404 ? new TRPCError({ code: "NOT_FOUND", message: err.message })
  : err?.code === 504 ? new TRPCError({ code: "TIMEOUT", message: err.message })
  : new TRPCError({ code: "INTERNAL_SERVER_ERROR", message: err?.message ?? String(err) });

export const gatewayRouter = t.router({
  products: t.router({
    get: t.procedure
      .input(z.object({ id: z.number().int() }))
      .query(({ input }) => broker.call<Product, { id: number }>("products.get", input).catch((e) => { throw toTRPCError(e); })),
    list: t.procedure
      .input(z.object({ limit: z.number().int().default(10) }))
      .query(({ input }) => broker.call<Product[], { limit: number }>("products.list", input)),
  }),
});
export type GatewayRouter = typeof gatewayRouter;

await broker.start();
await broker.waitForServices("products");
createHTTPServer({ router: gatewayRouter }).listen(4010);

The tRPC client is the one from the first section with the router type swapped:

[trpc-gateway] tRPC gateway on :4010, backed by products-1, products-2
  call 1: Keyboard served by products-1
  call 2: Keyboard served by products-2
  call 3: Keyboard served by products-1
  call 4: Keyboard served by products-2
  id 99: NOT_FOUND (HTTP 404) — product 99 not found

One URL for the browser, two instances behind it, no proxy — the balancing moved from where tRPC cannot do it to where Moleculer does it by default. (moleculer-web does the same job for REST/GraphQL clients; tRPC is the choice when the consumer is your own TypeScript frontend.)

A gRPC service behind a Moleculer action

The other direction: an inventory service written in Go — stood in for by a Node gRPC server here, nothing on the Moleculer side depends on that — that the rest of the cluster should be able to call like any other action.

// combined/inventory.service.js
module.exports = {
  name: "inventory",
  settings: { grpcAddress: process.env.INVENTORY_GRPC || "127.0.0.1:50060" },

  actions: {
    check: {
      params: { sku: "string" },
      handler(ctx) {
        return new Promise((resolve, reject) => {
          // Turn the Moleculer timeout budget into a gRPC deadline, so the remote side stops too.
          const opts = ctx.options.timeout > 0 ? { deadline: Date.now() + ctx.options.timeout } : {};
          this.client.Check({ sku: ctx.params.sku }, opts, (err, res) =>
            err ? reject(this.mapError(err, ctx)) : resolve({ sku: res.sku, inStock: res.in_stock, warehouse: res.warehouse }));
        });
      },
    },
  },

  methods: {
    // gRPC status → Moleculer error, so callers see the same error shapes as from any other service
    mapError(err, ctx) {
      switch (err.code) {
        case grpc.status.NOT_FOUND:         return new Errors.MoleculerClientError(err.details, 404, "NOT_FOUND", { sku: ctx.params.sku });
        case grpc.status.DEADLINE_EXCEEDED: return new Errors.RequestTimeoutError({ action: "inventory.check (gRPC)", nodeID: this.settings.grpcAddress });
        case grpc.status.UNAVAILABLE:       return new Errors.MoleculerRetryableError(`inventory gRPC backend unavailable: ${err.details}`, 503, "UNAVAILABLE");
        default:                            return new Errors.MoleculerError(err.details || err.message, 500, `GRPC_${grpc.status[err.code]}`);
      }
    },
  },

  created() { this.client = new pkg.Inventory(this.settings.grpcAddress, grpc.credentials.createInsecure()); },
  stopped() { this.client.close(); },
};

Any node in the cluster calls inventory.check; only this file knows gRPC exists. The backend answers in 200 ms in the demo:

  SKU-1: {"sku":"SKU-1","inStock":42,"warehouse":"WH-EU-1"} (268 ms)
  SKU-9: MoleculerClientError (404 NOT_FOUND) — unknown sku SKU-9 (217 ms)
  SKU-1, timeout 150 ms vs a slow backend: RequestTimeoutError (504 REQUEST_TIMEOUT) — Request is timed out when call 'inventory.check' action on 'inventory-1' node. (153 ms)
--- inventory-1 log:
  WARN  INVENTORY: gRPC deadline hit for sku SKU-1 (budget was 150 ms)

The third line is the one to notice: the caller's 150 ms timeout became a 150 ms gRPC deadline, the Go side gave up at the same moment, and the wrapper logged it. The UNAVAILABLE → MoleculerRetryableError mapping means a restart of the Go service is retried by the cluster's retry policy, and the breaker counts real failures — the resilience layer now covers a service that was never written for it.

Which one, when

Your frontend is TypeScript and calls your own backend — tRPC, no contest. If that backend is a cluster of services, put tRPC on the gateway node and let Moleculer do the rest, as above.
Services in different languages must call each other, or you need bidirectional streaming — gRPC for that boundary. Wrap it in a Moleculer service if the rest of your system is Node, so the boundary gets timeouts, retries and a breaker without every caller reimplementing them.
A fleet of Node.js services that need to find each other, balance, fail over and publish events — and you do not want to run a mesh to get it — that is what Moleculer is, and the RPC inside it is the least interesting part. Accept the weaker static typing, or fix it at the edges with tRPC.
Already on Kubernetes with a mesh doing discovery, LB and retries — gRPC is a natural fit and Moleculer's registry is redundant; you would use it for events, caching and the programming model, or not at all. (The Kubernetes post argues the other case.)

The unhelpful framing is "which RPC is fastest". Problem 3 above is what a real outage looks like, and the interesting question is how much of the answer to it your tool ships with versus how much you are about to write.

Code for this post: examples/08-vs-grpc-trpc — run.sh reproduces every demo against a local NATS container. @grpc/grpc-js 1.14.4, @trpc/server 11.18, zod 4.5, Moleculer 0.15.2, Node 22.

Previous in the series: Event-driven architecture in Node.js: pub/sub with NATS, Kafka and Redis. This is the last post of the series.

Event-driven architecture in Node.js: pub/sub with NATS, Kafka and Redis — and when an event is not enough

Icebob — Tue, 13 Oct 2026 13:00:34 +0000

Request/response is how a microservice system starts, and events are how it stops being a distributed monolith. The moment orders has to call mailer, analytics and inventory after every checkout, orders knows about three services it should not care about, is as slow as the slowest of them, and fails when any of them is down. Flip it around — orders announces "order created", and whoever cares subscribes — and the coupling goes away.

Then two questions show up that every pub/sub design has to answer, and that most tutorials skip: who exactly gets the event when a subscriber runs in three copies, and what happens to it when nobody is listening. This post answers both with running code, on three different message brokers, and ends with the part where a plain event is the wrong tool and you need something with an acknowledgement.

Every snippet was executed; the logs are pasted, not typed. The code is in the example repo (run.sh reproduces all five demos).

If you are new to Moleculer: a service is a plain object with actions (request/response) and events (pub/sub handlers); a broker runs services and connects nodes over a transporter — the message broker in the middle, which can be NATS, Redis, Kafka, MQTT, AMQP or plain TCP. broker.emit("order.created", payload) is the publish side; an events: { "order.created"(ctx) {} } block is the subscribe side. The routing logic — which subscriber instance gets what — is Moleculer's own and is the same on every transporter, which is what makes the "same code, three brokers" section below possible.

The setup: one producer, two kinds of consumers

orders creates orders and announces them. Two subscribers: mailer (which we will run twice, to see load balancing) and analytics (which listens with a wildcard).

// services/orders.service.js
module.exports = {
  name: "orders",
  created() { this.seq = 0; },

  actions: {
    create: {
      params: { sku: "string", qty: { type: "number", default: 1 } },
      async handler(ctx) {
        const order = { id: ++this.seq, sku: ctx.params.sku, qty: ctx.params.qty, total: ctx.params.qty * 10 };
        log(this, `orders.create #${order.id} (${order.sku}) — requestID ${ctx.requestID.slice(0, 8)}, level ${ctx.level}`);
        // event: delivered to ONE instance of every subscriber service that is online right now
        await ctx.emit("order.created", order);
        return order;
      },
    },
    cancel: {
      params: { id: "number" },
      async handler(ctx) {
        // broadcast: delivered to EVERY instance of every subscriber service
        await ctx.broadcast("order.cancelled", { id: ctx.params.id });
        return { id: ctx.params.id, cancelled: true };
      },
    },
  },
};

// services/mailer.service.js
module.exports = {
  name: "mailer",
  events: {
    "order.created": {
      // event payloads can be validated like action params (same fastest-validator schema)
      params: { id: "number", sku: "string", total: "number" },
      async handler(ctx) {
        const { id, sku } = ctx.params;
        log(this, `order.created #${id} (${sku}) — requestID ${ctx.requestID.slice(0, 8)}, meta.user ${ctx.meta.user}, level ${ctx.level}`);
        // events can chain: the context carries the same requestID one level deeper
        await ctx.emit("email.sent", { orderId: id, to: `${ctx.meta.user}@example.com` });
      },
    },
    "order.cancelled"(ctx) {
      log(this, `order.cancelled #${ctx.params.id} — dropping queued emails for it`);
    },
  },
};

// services/analytics.service.js
module.exports = {
  name: "analytics",
  events: {
    "order.*"(ctx) {
      log(this, `${ctx.eventName} #${ctx.params.id} — ${ctx.eventType} from ${ctx.nodeID}, level ${ctx.level}`);
    },
    "email.sent"(ctx) {
      log(this, `email.sent for #${ctx.params.orderId} to ${ctx.params.to} — requestID ${ctx.requestID.slice(0, 8)}, level ${ctx.level}`);
    },
  },
};

(log prefixes the line with a timestamp and the node ID so the logs of four processes can be merged in order — that is all it does.)

The config is two lines that matter. The transporter is an environment variable because we are going to swap it:

// moleculer.config.js
module.exports = {
  nodeID: process.env.NODE_NAME || `node-${process.pid}`,
  transporter: process.env.TRANSPORTER || "nats://localhost:4222",
  // "redis://localhost:6379"  or  "kafka://localhost:9092" — nothing else changes
};

Four nodes, each started with moleculer-runner and a SERVICES= filter: app-1 (orders), mailer-1, mailer-2, analytics-1. A fifth throwaway broker (client.js) calls orders.create four times with meta: { user: "alice" }, then orders.cancel once.

1. `emit` is balanced, `broadcast` is not

--- 4× orders.create (each one emits order.created)
[app-1] orders.create #1 (SKU-1) — requestID 3b5d9c00, level 1
[analytics-1] order.created #1 — emit from app-1, level 2
[mailer-2] order.created #1 (SKU-1) — requestID 3b5d9c00, meta.user alice, level 2
[analytics-1] email.sent for #1 to alice@example.com — requestID 3b5d9c00, level 3
[app-1] orders.create #2 (SKU-2) — requestID 128071b0, level 1
[analytics-1] order.created #2 — emit from app-1, level 2
[mailer-1] order.created #2 (SKU-2) — requestID 128071b0, meta.user alice, level 2
[analytics-1] email.sent for #2 to alice@example.com — requestID 128071b0, level 3
[app-1] orders.create #3 (SKU-3) — requestID ab6b4a4f, level 1
[analytics-1] order.created #3 — emit from app-1, level 2
[mailer-2] order.created #3 (SKU-3) — requestID ab6b4a4f, meta.user alice, level 2
[analytics-1] email.sent for #3 to alice@example.com — requestID ab6b4a4f, level 3
[app-1] orders.create #4 (SKU-4) — requestID e1966cc6, level 1
[mailer-1] order.created #4 (SKU-4) — requestID e1966cc6, meta.user alice, level 2
[analytics-1] order.created #4 — emit from app-1, level 2
[analytics-1] email.sent for #4 to alice@example.com — requestID e1966cc6, level 3
--- orders.cancel #2 (broadcasts order.cancelled)
[mailer-2] order.cancelled #2 — dropping queued emails for it
[analytics-1] order.cancelled #2 — broadcast from app-1, level 2
[mailer-1] order.cancelled #2 — dropping queued emails for it

Three things in this log, in order of how often they surprise people.

Each order.created reached one mailer, not two. emit is a balanced event: every subscribing service gets the event once, and if that service runs in several instances, the emitter's registry picks one — mailer-2, mailer-1, mailer-2, mailer-1: round-robin, the same strategy the load balancer uses for action calls. analytics is a different service, so it gets its own copy of every event. In broker vocabulary each service name is an event group, and emit delivers one copy per group. This is what a queue-group or consumer-group gives you in NATS or Kafka, except it works identically on all transporters because the choice is made by the emitting broker, not by the message broker. (One consequence: if the emitting node also hosts an instance of the subscriber, that local instance always wins — registry.preferLocal applies to events too.)

order.cancelled reached both mailers. broadcast goes to every instance of every subscriber. That is the right verb for "flush your cache", "reload config", "a node joined" — anything where each process has to react. For "an order was created, someone send the email", it would send two emails. There is also broadcastLocal, which only reaches services in the same process, and it is what the broker's own $node.connected/$services.changed events use.

The request ID survived two hops and the user survived with it. orders.create ran at level 1 with request 3b5d9c00; the mailer's handler ran at level 2 with the same request ID and the same meta.user that the client put on the action call; the email.sent event that the mailer emitted from inside its handler arrived at analytics at level 3, still 3b5d9c00. Event handlers receive a full Context — ctx.params, ctx.meta, ctx.requestID, ctx.call and ctx.emit — so an event is not a dead end for tracing or for tenant/user information. Enable the tracing middleware and this chain shows up as one trace with three spans, without passing anything by hand.

2. Groups and wildcards

analytics subscribed to order.* and received both order.created and order.cancelled, with the actual name in ctx.eventName. Wildcards are * for one segment and ** for any number — ** is how you write an audit service that sees everything.

The group can also be targeted from the emitting side. This emits order.created to the mailer group only:

await broker.emit("order.created", { id: 99, sku: "SKU-99", total: 0 }, ["mailer"]);

=== 2. Event groups: emit only to the mailer group ===
[mailer-1] order.created #99 (SKU-99) — requestID dccf0532, meta.user undefined, level 1
[analytics-1] email.sent for #99 to undefined@example.com — requestID dccf0532, level 2

analytics never saw order.created #99 — it only saw the email.sent that the mailer emitted afterwards, which had no group filter. This is handy for replays ("re-send order 99 to the mailer, but do not count it again") and for the reverse case, where a service wants to join another service's group: an event definition can say group: "mailer", and then that handler and the real mailer share one round-robin — only one of them gets each event.

3. Validation happens at the subscriber

The mailer declared a params schema on its handler. What happens when a producer sends garbage?

await broker.emit("order.created", { id: "oops" });

=== 3. Event payload validation ===
[analytics-1] order.created #oops — emit from client-3557966, level 1
--- and on the mailer node that rejected it:
[19:39:50.367Z] ERROR TRANSIT: ValidationError: Parameters validation error!
    at processCheckResponse (…/node_modules/moleculer/src/validators/base.js:103:6)
  data: [ { type: 'number', message: "The 'id' field must be a number.", field: 'id', actual: 'oops', nodeID: 'client-3557966', event: 'order.created' }, { type: 'required', message: "The 'sku' field is required.", field: 'sku', … }, { type: 'required', message: "The 'total' field is required.", field: 'total', … } ]

Two points. The schema is enforced per handler: analytics had no schema on its wildcard handler, so it happily logged #oops. And the error was logged on the mailer node and went nowhere else — broker.emit had already resolved on the client. That is the contract of an event: emit resolves when the event has been handed to the transporter, not when it has been handled. A handler that throws — validation or otherwise — is logged by the receiving broker and that is the end of it. If the producer needs to know the outcome, it wanted an action call, not an event. Keep that sentence in mind for section 5.

4. Same code, three brokers

Nothing in the services above mentions NATS. So let's run the same bench pair — one node emitting 2 000 small events, one node subscribed, counting arrival latency — on all three transporters, changing only TRANSPORTER=:

=== 4. Same code, three transporters: 2 000 events, one emitter node → one subscriber node ===
--- TRANSPORTER=nats
  awaited each emit: 2000 events in 346 ms (5780 events/s) — received 2000, latency p50 2 ms, p99 8 ms, max 9 ms
  fire-and-forget: 2000 events in 305 ms (6557 events/s) — received 2000, latency p50 29 ms, p99 72 ms, max 72 ms
--- TRANSPORTER=redis
  awaited each emit: 2000 events in 287 ms (6969 events/s) — received 2000, latency p50 2 ms, p99 14 ms, max 16 ms
  fire-and-forget: 2000 events in 249 ms (8032 events/s) — received 2000, latency p50 3 ms, p99 20 ms, max 20 ms
--- TRANSPORTER=kafka
  awaited each emit: 2000 events in 5983 ms (334 events/s) — received 2000, latency p50 4 ms, p99 19 ms, max 96 ms
  fire-and-forget: 2000 events in 2105 ms (950 events/s) — received 2000, latency p50 1186 ms, p99 1936 ms, max 1955 ms

("Awaited each emit" is a loop of await broker.emit(...); "fire-and-forget" fires all 2 000 and then awaits the promises, so its latency numbers include queueing behind the previous 1 999. Single machine, brokers in Docker, one run — treat these as shapes, not benchmarks; the NATS and Redis numbers moved by ±50 % between runs.)

The shapes are the story. NATS and Redis are in the same class: a few thousand events per second from a single emitter loop, single-digit-millisecond delivery. Kafka is 10–20× slower on this path — and this is not Kafka being slow, it is Kafka being used for something it is not built for. Here is what each transporter actually does with an event, which explains both the numbers and the durability question coming next:

NATS (nats://) — core NATS pub/sub. Each node subscribes to its own subject (MOL.EVENT.<nodeID>); the emitter's registry decides the target node and publishes there. At-most-once, no persistence, and the fastest of the three. NATS also has queue groups, and if you set disableBalancer: true Moleculer stops picking the node itself and lets NATS balance within the group instead — the only one of the three transporters that supports this.
Redis (redis://) — Redis Pub/Sub (PUBLISH/SUBSCRIBE), one channel per node. Same semantics as NATS: a message published while no subscriber is connected is dropped; there is no queue behind it. Nearly as fast, and often already in the stack.
Kafka (kafka://) — one topic per node (MOL.EVENT.<nodeID>, auto-created), every message to partition 0, and every Moleculer process is its own consumer group (groupId = the broker's unique instance ID). Kafka's own consumer-group balancing and offset replay are deliberately not in play — Moleculer already balanced the event in its registry and targeted a node, so the topic is just a pipe to that node. Each emit is a producer round trip that waits for the broker's ack, which is where the ~3 ms per event goes.

So if you were about to pick Kafka as the transporter because "we need durability", stop: the transporter is a transport. It gets a packet from node A to node B; the registry, the load balancing and the event routing sit above it and behave the same on all of them. Pick the transporter for operational reasons — what you already run, what your ops team can keep alive, latency — and get durability from the layer that is built for it. Which brings us to the demo that matters most.

5. What happens when the subscriber is down

The scenario every event-driven system meets on its first deploy: orders is up, mailer is being restarted, two orders come in.

=== 5. What happens when the subscriber is down ===
--- events: orders runs, mailer is NOT running; create 2 orders, then start the mailer
  → orders.create returned #1
  → orders.create returned #2
--- mailer log:
(nothing — the two events were delivered to nobody and are gone)

emit resolved happily both times. The registry had no mailer endpoint online, so there was no one to send to, so it sent to no one. The mailer came back three seconds later and got nothing, because nothing was stored anywhere — not in NATS, not in Redis, and (see above) not in a way Kafka would replay either. This is at-most-once delivery, and for a lot of events it is exactly right: a cache-invalidation, a "user is typing" indicator, a metric sample, a UI push. Losing one costs nothing, and the source of truth can be re-read.

An order confirmation email is not in that category. Neither is a payment webhook, an inventory reservation, or anything that ends up in an audit log. For these you need a message that is stored until acknowledged, retried when the handler throws, and parked somewhere visible when it keeps failing. In Moleculer that is a separate module — @moleculer/channels — with adapters for Redis Streams, Kafka, NATS JetStream and AMQP. It is one middleware in the config:

// moleculer.config.js (with CHANNELS=1)
const { Middleware: ChannelsMiddleware } = require("@moleculer/channels");

module.exports = {
  transporter: "nats://localhost:4222",   // still the transporter for calls and events
  middlewares: [
    ChannelsMiddleware({
      adapter: {
        type: "Redis",
        options: {
          maxRetries: 3,                                              // then dead-letter (or drop, if dead-lettering is off)
          deadLettering: { enabled: true, queueName: "FAILED_MESSAGES" },
          redis: {
            url: "redis://localhost:6379",
            consumerOptions: { minIdleTime: 1000, claimInterval: 200, processingAttemptsInterval: 500 },
            // minIdleTime: an unacked message is retried after 1 s (default: 1 hour)
          },
        },
      },
    }),
  ],
};

The producer change is one line — sendToChannel instead of emit:

// services/orders.service.js — create handler
if (process.env.CHANNELS === "1") {
  await this.broker.sendToChannel("order.created", order); // stored in the stream until a consumer acknowledges it
} else {
  await ctx.emit("order.created", order);
}

And the consumer gets a channels block next to events. The handler receives the raw payload — no Moleculer context, because the message may have been written to the stream by anything, not just a broker (set context: true if you want ctx.meta and tracing carried through headers):

// services/mailer.service.js
channels: {
  "order.created": {
    async handler(payload) {
      log(this, `[channel] order.created #${payload.id} (${payload.sku})`);
      if (payload.id === FAIL_ORDER) throw new Error(`SMTP is down, could not mail order #${payload.id}`);
    },
  },
},

Plus a support service that consumes the dead-letter queue, with context: true so the error the adapter attached is readable from ctx.headers:

// services/support.service.js
channels: {
  FAILED_MESSAGES: {
    context: true,
    async handler(ctx) {
      log(this, `[dead-letter] order #${ctx.params.id} — ${ctx.headers["x-error-name"]}: ${ctx.headers["x-error-message"]}`);
    },
  },
},

Same scenario, with a twist at the end: the mailer is started once (so its consumer group exists), then killed; two orders are created while it is down; it comes back; then order #3 is created, which this mailer instance was told to fail (FAIL_ORDER=3).

--- channels (CHANNELS=1, Redis Streams): same scenario
mailer-1 stopped
  → orders.create returned #1
  → orders.create returned #2
messages waiting in the stream: 2
--- order #3 makes the mailer throw: retries, then dead-letter
  → orders.create returned #3
[app-1] orders.create #1 (SKU-1) — requestID bf278ea3, level 1
[app-1] orders.create #2 (SKU-2) — requestID a68bf01b, level 1
[mailer-1] [channel] order.created #1 (SKU-1)
[mailer-1] [channel] order.created #2 (SKU-2)
[app-1] orders.create #3 (SKU-1) — requestID a7031da1, level 1
[mailer-1] [channel] order.created #3 (SKU-1)
[mailer-1] [channel] order.created #3 (SKU-1)
[mailer-1] [channel] order.created #3 (SKU-1)
[support-1] [dead-letter] order #3 — Error: SMTP is down, could not mail order #3
WARN  CHANNELS: Moved message to 'FAILED_MESSAGES' 1788550869933-0

Orders #1 and #2 sat in the Redis stream (XLEN said 2) and were delivered the moment the mailer's consumer reconnected. Order #3 was attempted three times, a second apart — the handler threw, the message was never acknowledged, and after minIdleTime the adapter claimed it back and redelivered it — and after maxRetries it was moved to FAILED_MESSAGES, where the support service picked it up with the original payload and the error message in headers. Multiple mailer instances would share the consumer group, so this also load-balances, like emit did, but with the stream keeping score.

One trap worth knowing before your first outage: Redis Streams consumer groups start reading at $ (new messages only) by default, so a subscriber that has never run does not get messages sent before its first start. Deploy consumers before producers, or set redis.startID: "0" on the channel. Kafka and JetStream adapters have the equivalent knobs (fromBeginning, stream config).

Which one, when

The framework gives you two verbs with different guarantees, and the mistake is treating them as one:

emit / broadcast (events) — routed by the registry to whoever is online now, one copy per service (or per instance), at-most-once, transporter-agnostic, sub-10 ms. Use them to decouple services when losing an occasional message is acceptable or recoverable: cache invalidation, live updates, metrics, "something changed, go re-read it". This covers more of a typical system than people expect.
sendToChannel (channels) — stored in Redis Streams / Kafka / JetStream / AMQP until acknowledged, retried, dead-lettered, consumer-group balanced. Use it for anything with a cost of loss: emails, payments, stock movements, outbox patterns, integration with non-Moleculer producers and consumers.
The transporter is neither of these. It is the wire between nodes for calls and events, and its choice does not change what emit guarantees. NATS if you can choose; Redis if it is already there; Kafka as a transporter only when you have a reason that is not "durability" — for durability use it as a channel adapter instead.

If you take one thing from the logs: in section 1 the mailer got every event, in section 5 it got none, and the code was identical. The difference was whether it was running. Design your event handlers for that — and reach for a channel when you cannot.

Code for this post: examples/07-event-driven — run.sh reproduces all five demos against local NATS, Redis and Kafka containers. Moleculer 0.15.2, @moleculer/channels 0.3.1, Node 22.

Previous in the series: Circuit breaker, retry and timeout in Node.js microservices. Next: Moleculer vs gRPC and tRPC — RPC transport vs. a full service layer.

Circuit breaker, retry and timeout in Node.js microservices — and why you need all three

Icebob — Tue, 06 Oct 2026 13:00:19 +0000

A microservice architecture has one failure mode that a monolith never had: a dependency that is almost working. Not down — down is easy, the connection is refused and you know in a millisecond. Almost working means it answers one call in three, or it answers everything but takes four seconds to do it. await has no opinion about that. It waits. Your request handler waits with it, the HTTP client in front of you waits, and after enough of these pile up, the service that was perfectly healthy a minute ago is now also "almost working".

The standard answer is three guards that every mature RPC stack has some version of: a timeout (stop waiting), a retry policy (try again when it makes sense) and a circuit breaker (stop trying when it doesn't). In Node.js you usually get these from a library you wrap around each call — cockatiel, opossum, or p-retry plus something home-grown. In Moleculer they are broker options: one config block, applied to every service-to-service call in the cluster, with per-action overrides when one endpoint needs different rules.

This post switches each of them on, one at a time, against a dependency that we break on purpose, and shows the actual output. Every snippet below was run; the logs are pasted, not typed. The runnable code is in the example repo (run.sh reproduces every demo).

If you are new to Moleculer: a service is a plain object with actions (callable functions), and a broker runs services and routes calls between nodes over a transporter (NATS here). broker.call("inventory.check", params) is a request/response RPC that may land in-process or on another machine. The service discovery post covers how the broker knows where inventory.check lives.

The setup: a dependency with knobs

Three services. inventory is the one we abuse — it has runtime knobs so a single instance can be made slow or flaky while its siblings stay healthy:

// services/inventory.service.js
const { Errors } = require("moleculer");

module.exports = {
  name: "inventory",

  created() {
    this.knobs = {
      failRate: Number(process.env.FAIL_RATE || 0),   // 0..1 — share of calls that throw
      latency: Number(process.env.LATENCY_MS || 0),   // ms — artificial slowness
      failEvery: Number(process.env.FAIL_EVERY || 0), // 2 = every 2nd call fails (deterministic)
    };
    this.calls = 0;
  },

  actions: {
    check: {
      params: { sku: "string" },
      async handler(ctx) {
        this.calls++;
        const { failRate, latency, failEvery } = this.knobs;
        if (latency) await new Promise((r) => setTimeout(r, latency));
        const fail = (failEvery && this.calls % failEvery === 0) || Math.random() < failRate;
        if (fail) {
          // 503 + retryable: "try again, it might work"
          throw new Errors.MoleculerRetryableError("inventory database unavailable", 503, "DB_UNAVAILABLE");
        }
        return { sku: ctx.params.sku, inStock: 42, servedBy: this.broker.nodeID };
      },
    },

    configure: {
      params: { failRate: { type: "number", optional: true }, latency: { type: "number", optional: true }, failEvery: { type: "number", optional: true } },
      handler(ctx) {
        Object.assign(this.knobs, ctx.params);
        return this.knobs;
      },
    },
  },
};

orders.create calls inventory.check and then shipping.quote in sequence (we need a two-hop chain for the timeout section), and shipping.quote can be slowed down with SHIPPING_LATENCY_MS. Nothing else in them is interesting.

The config is shared by every node, and the three guards are switched on with environment variables so that each demo can show the same code with the feature off and then on:

// moleculer.config.js
module.exports = {
  nodeID: process.env.NODE_NAME || `node-${process.pid}`,
  transporter: process.env.TRANSPORTER || "nats://localhost:4222",

  requestTimeout: Number(process.env.REQUEST_TIMEOUT_MS || 0), // 0 = no timeout (the default!)

  retryPolicy: {
    enabled: process.env.RETRY === "1",
    retries: 3,
    delay: 100,      // first wait; then ×factor, capped at maxDelay
    maxDelay: 1000,
    factor: 2,
    check: (err) => err && !!err.retryable, // only errors that say "try again" — never a 4xx
  },

  circuitBreaker: {
    enabled: process.env.CB === "1",
    threshold: 0.5,        // open when ≥ 50 % of the window failed…
    minRequestCount: 5,    // …but only after at least 5 calls (default 20)
    windowTime: 60,        // seconds the counters are kept
    halfOpenTime: 5 * 1000, // ms until a single probe call is let through (default 10 s)
    check: (err) => err && err.code >= 500, // what counts as a failure
  },

  logger: { type: "Console", options: { level: process.env.LOG_LEVEL || "warn", formatter: "short" } },
};

Two things in that block are worth saying out loud before we run anything. All three guards are off by default, and requestTimeout defaults to 0, which means no timeout at all. A fresh Moleculer project will wait forever for a dead node, exactly like a fresh fetch() will. The framework gives you the tools; it does not guess your SLOs for you.

Workers are started with the standard runner, one process per node, e.g. NODE_NAME=inventory-bad FAIL_RATE=1 SERVICES=inventory SERVICEDIR=services npx moleculer-runner --config moleculer.config.js. The client (client.js) is a broker with no services that joins the same NATS and calls inventory.check in a loop, printing which node answered and how long it took.

1. Retry: try again, but only when it makes sense

inventory-1 runs with FAIL_EVERY=2 — every second call throws the retryable 503. Ten calls, retry policy off:

--- no retry policy (the default)
  ✓ inventory-1 inStock=42 (7 ms)
  ✗ MoleculerRetryableError (503) @inventory-1: inventory database unavailable (5 ms)
  ✓ inventory-1 inStock=42 (2 ms)
  ✗ MoleculerRetryableError (503) @inventory-1: inventory database unavailable (2 ms)
  ✓ inventory-1 inStock=42 (4 ms)
  ✗ MoleculerRetryableError (503) @inventory-1: inventory database unavailable (4 ms)
  ✓ inventory-1 inStock=42 (3 ms)
  ✗ MoleculerRetryableError (503) @inventory-1: inventory database unavailable (4 ms)
  ✓ inventory-1 inStock=42 (4 ms)
  ✗ MoleculerRetryableError (503) @inventory-1: inventory database unavailable (4 ms)

Fifty percent error rate, as designed. Same workers, same client, RETRY=1:

--- RETRY=1 (retries: 3, delay: 100 ms, factor 2)
  ✓ inventory-1 inStock=42 (28 ms)
[21:02:08.093Z] WARN  BROKER: Retry to call 'inventory.check' action after 100 ms... { requestID: 'aeba50eb-…', attempts: 1 }
  ✓ inventory-1 inStock=42 (122 ms)
[21:02:08.220Z] WARN  BROKER: Retry to call 'inventory.check' action after 100 ms... { requestID: 'b4405bb3-…', attempts: 1 }
  ✓ inventory-1 inStock=42 (120 ms)
[21:02:08.337Z] WARN  BROKER: Retry to call 'inventory.check' action after 100 ms... { requestID: '533175a5-…', attempts: 1 }
  ✓ inventory-1 inStock=42 (116 ms)
  … (same for the remaining six)

Ten out of ten succeed. Nine of them were retried once (the first call happened to hit an odd count, after that every call's first attempt lands on an even one), each paying the 100 ms delay — visible in the timings. Nothing in the service or the client changed; the retry lives in the broker of the caller, which is the only place it can live, because the callee cannot retry itself.

Three details that matter more than the happy path:

Which errors get retried is a property of the error, not of the policy. The default check is err => err.retryable. MoleculerRetryableError (and MoleculerServerError, which extends it) set that flag; the plain MoleculerError and every validation/4xx error do not. So a ValidationError is never retried no matter what you configure, and a 503 from a database that might come back is. When you throw from your own code, you decide — new Errors.MoleculerRetryableError(msg, 503, "DB_UNAVAILABLE") versus new Errors.MoleculerError(msg, 409, "DUPLICATE"). Think of it as the callee telling the caller whether trying again could possibly help.

The backoff is exponential and capped: delay, then delay × factor, then × factor again, never above maxDelay. With the values above that is 100 → 200 → 400 ms for the three retries, so a call that never succeeds costs about 700 ms of waiting on top of the attempts themselves. Set maxDelay with that arithmetic in mind.

Retries are per call, and they change which node answers. The retry goes through endpoint selection again, so with two instances of inventory a retry after a failure on one of them will typically land on the other. That is the failover behaviour you want — and the reason retries only make sense on idempotent actions. inventory.check can be run twice with no harm; payments.charge cannot. For those, either leave retries off for that action or make the action itself idempotent with a request key. Per-action and per-call overrides exist for exactly this:

// per action — in the service schema
actions: {
  charge: {
    retryPolicy: { enabled: false },
    async handler(ctx) { /* … */ },
  },
}

// per call — in the calling code
await ctx.call("inventory.check", { sku }, { retries: 5 });

2. Circuit breaker: stop asking a node that keeps saying no

Two inventory nodes now: inventory-good and inventory-bad (FAIL_RATE=1, fails everything). Round-robin sends every second call to the broken one, and without a breaker nothing ever changes that:

--- no circuit breaker: the broken node keeps getting every 2nd call
  ✗ # 1 MoleculerRetryableError (503) @inventory-bad: inventory database unavailable (14 ms)
  ✓ # 2 inventory-good inStock=42 (6 ms)
  ✗ # 3 MoleculerRetryableError (503) @inventory-bad: inventory database unavailable (6 ms)
  ✓ # 4 inventory-good inStock=42 (7 ms)
  ✗ # 5 MoleculerRetryableError (503) @inventory-bad: inventory database unavailable (10 ms)
  ✓ # 6 inventory-good inStock=42 (37 ms)
  … (alternating like this for all 12)

With CB=1 on the client. The client also registers listeners for the three $circuit-breaker.* events the broker broadcasts, printed as coloured dots. After the 12th call the script "repairs" the bad node with a targeted call (broker.call("inventory.configure", { failRate: 0 }, { nodeID: "inventory-bad" }) — nodeID in the call options bypasses load balancing) and keeps calling every 600 ms:

--- CB=1: the breaker isolates inventory-bad, then we fix it and it comes back
  ✓ # 1 inventory-good inStock=42 (11 ms)
  ✗ # 2 MoleculerRetryableError (503) @inventory-bad: inventory database unavailable (29 ms)
  ✓ # 3 inventory-good inStock=42 (5 ms)
  ✗ # 4 MoleculerRetryableError (503) @inventory-bad: inventory database unavailable (12 ms)
  ✓ # 5 inventory-good inStock=42 (6 ms)
  ✗ # 6 MoleculerRetryableError (503) @inventory-bad: inventory database unavailable (6 ms)
  ✓ # 7 inventory-good inStock=42 (6 ms)
  ✗ # 8 MoleculerRetryableError (503) @inventory-bad: inventory database unavailable (5 ms)
  ✓ # 9 inventory-good inStock=42 (9 ms)
  🔴 circuit OPENED   for inventory.check on inventory-bad (5/5 failed)
  ✗ #10 MoleculerRetryableError (503) @inventory-bad: inventory database unavailable (5 ms)
  ✓ #11 inventory-good inStock=42 (7 ms)
  ✓ #12 inventory-good inStock=42 (6 ms)
  → inventory.configure on inventory-bad: failRate=0 (the node is "fixed")
  ✓ #13 inventory-good inStock=42 (3 ms)
  ✓ #14 inventory-good inStock=42 (4 ms)
  ✓ #15 inventory-good inStock=42 (5 ms)
  ✓ #16 inventory-good inStock=42 (5 ms)
  ✓ #17 inventory-good inStock=42 (4 ms)
  ✓ #18 inventory-good inStock=42 (4 ms)
  ✓ #19 inventory-good inStock=42 (5 ms)
  ✓ #20 inventory-good inStock=42 (4 ms)
  🟡 circuit HALF-OPEN for inventory.check on inventory-bad — one probe call allowed
  🟢 circuit CLOSED   for inventory.check on inventory-bad — back in rotation
  ✓ #21 inventory-bad inStock=42 (6 ms)
  ✓ #22 inventory-good inStock=42 (17 ms)
  ✓ #23 inventory-bad inStock=42 (12 ms)
  ✓ #24 inventory-good inStock=42 (4 ms)

Reading it top to bottom:

Calls 2–10 fail on inventory-bad. The fifth failure (call #10) satisfies minRequestCount: 5 with a 100 % failure rate, above the threshold: 0.5, so the breaker opens — the event fires while call #10 is still returning its error.
From #11 on, inventory-bad gets nothing. Not because the client checks a flag before each call: the breaker sets the endpoint's state to unavailable in the local registry, so the load balancer simply does not see it any more. Every call goes to inventory-good. The error rate drops from 50 % to zero without touching the broken node.
We fix the node at #12. Nothing changes immediately — the breaker does not know, and that is correct. Five seconds after it opened (halfOpenTime) it goes half-open: the endpoint is made visible again for exactly one call, and hidden again until that probe completes.
The probe (#21) succeeds, the breaker closes, and round-robin resumes alternating. Had the probe failed, the breaker would have gone straight back to open for another 5 s.

Two properties that are easy to get wrong when you build this yourself, and that the broker gets right because it is the load balancer:

The breaker is per endpoint — per (node, action) pair — not per service. inventory.check on inventory-bad was open while inventory.check on inventory-good was closed. A breaker per service name would have taken the healthy node down with the broken one. A breaker per node would have blocked inventory.configure too, and we could not have repaired it through the same channel.

Failure counting happens on the caller, and the caller decides what a failure is. The default check is err.code >= 500: server errors and timeouts count, validation errors and not-founds do not. That is per caller; a caller with different needs (a batch job that treats slow responses as failures) can register its own check. There is also a wrinkle in the source worth knowing: an error that originates on a third node (an inventory handler that itself called something that failed) is not counted against the inventory endpoint — only errors raised by the node you called, or locally, are.

Per-action overrides use the same shape as the broker option:

actions: {
  check: {
    circuitBreaker: { threshold: 0.3, minRequestCount: 10 },
    async handler(ctx) { /* … */ },
  },
}

And the $circuit-breaker.opened / half-opened / closed events are ordinary broker events — the payload has nodeID, action, failures, count, rate — so a monitoring service can subscribe to them like to anything else. If you have metrics enabled there are gauges for active open and half-open breakers as well.

3. Timeout: the one that is off by default

inventory-slow answers after 3 seconds. The client calls with a per-call timeout of 1 s:

--- only the slow node exists
[21:04:56.031Z] WARN  BROKER: Request 'inventory.check' is timed out. { requestID: 'be7093ec-…', nodeID: 'inventory-slow', timeout: 1000 }
  ✗ #1 timeout=1000 RequestTimeoutError (504) @inventory-slow: Request is timed out when call 'inventory.check' action on 'inventory-slow' node. (1007 ms)
  ✗ #2 timeout=1000 RequestTimeoutError (504) @inventory-slow: … (1011 ms)
  ✗ #3 timeout=1000 RequestTimeoutError (504) @inventory-slow: … (1005 ms)
  ✗ #4 timeout=1000 RequestTimeoutError (504) @inventory-slow: … (1003 ms)

That is all a timeout does on its own: it turns a 3-second wait into a 1-second RequestTimeoutError. Useful — your request handler is unblocked and can respond with a 504 — but the user still got nothing. Now look at the error's two properties: code 504 and retryable: true. That is where it composes with the previous two sections.

Start a healthy inventory-fast next to the slow one, and enable retry:

--- a healthy node joins; RETRY=1 turns the timeout into a failover
[21:05:04.302Z] WARN  BROKER: Request 'inventory.check' is timed out. { requestID: '50aa1ded-…', nodeID: 'inventory-slow', timeout: 1000 }
[21:05:04.305Z] WARN  BROKER: Retry to call 'inventory.check' action after 100 ms... { requestID: '50aa1ded-…', attempts: 1 }
  ✓ #1 timeout=1000 inventory-fast inStock=42 (1164 ms)
[21:05:05.451Z] WARN  BROKER: Request 'inventory.check' is timed out. { requestID: '7d3cfb02-…', nodeID: 'inventory-slow', timeout: 1000 }
[21:05:05.452Z] WARN  BROKER: Retry to call 'inventory.check' action after 100 ms... { requestID: '7d3cfb02-…', attempts: 1 }
  ✓ #2 timeout=1000 inventory-fast inStock=42 (1132 ms)
  … (#3, #4 identical)

Every call now succeeds — the timeout is retryable, the retry re-selects an endpoint, round-robin hands it to inventory-fast. But every call also costs 1.1 s, because round-robin keeps starting on the slow node. Retry is the wrong tool for a node that is consistently slow; it fixes the individual call and keeps paying the tax. Add the breaker:

--- RETRY=1 CB=1: timeouts count as failures, so the slow node gets cut off
  ✓ #1 timeout=1000 inventory-fast inStock=42 (11 ms)
  ✓ #2 timeout=1000 inventory-fast inStock=42 (1110 ms)
  ✓ #3 timeout=1000 inventory-fast inStock=42 (1108 ms)
  ✓ #4 timeout=1000 inventory-fast inStock=42 (1109 ms)
  ✓ #5 timeout=1000 inventory-fast inStock=42 (1109 ms)
  🔴 circuit OPENED   for inventory.check on inventory-slow (5/5 failed)
  ✓ #6 timeout=1000 inventory-fast inStock=42 (1108 ms)
  ✓ #7 timeout=1000 inventory-fast inStock=42 (6 ms)
  ✓ #8 timeout=1000 inventory-fast inStock=42 (7 ms)
  ✓ #9 timeout=1000 inventory-fast inStock=42 (7 ms)
  ✓ #10 timeout=1000 inventory-fast inStock=42 (5 ms)

Five timeouts (504 ≥ 500, so they count) open the breaker on inventory-slow, and from #7 on the calls are 5 ms. Nobody wrote any failover logic; three independent guards, each doing its narrow job, produced it. This composition is the actual point of the post — and it is also why the broker applies them in a fixed order, which matters:

Fallback is the outermost layer — it only kicks in after everything else has given up.
Retry wraps Timeout, so each attempt gets the full timeout; a retried call can take retries × timeout in the worst case. Budget accordingly.
Timeout wraps the Circuit breaker, so the caller's timeout fires outside the breaker. A slow endpoint is counted only when the callee's own timeout error (504) comes back through the breaker, and that happens only while retry keeps the request alive: with retries off, the caller drops the pending request first, the 504 arrives as an orphan response, and the breaker never sees it. Retry is on in this section, which is why the breaker opened above.
Bulkhead (section 6) is innermost and runs on the callee.

The timeout can be set on three levels, most specific wins: per call ({ timeout: 1000 } in the options, as above), per action (timeout: 1000 in the action schema — the callee saying "I should never take longer than this"), and per broker (requestTimeout). Set the broker-level one in every production config. A wrong value is better than 0.

4. Distributed timeout: the budget travels with the request

This is the part that a per-call timeout library cannot do for you. orders.create calls inventory.check (800 ms) and then shipping.quote (800 ms). The client calls orders.create with a timeout. First with a 3 s budget:

--- 3 s budget: fits
  ✓ orders.create timeout=3000 → inStock=42 timings={"inventory":813,"shipping":817} (1733 ms)

Then with 1.5 s, which is less than the two hops need:

--- 1.5 s budget: the second hop only gets what is left
[21:05:23.513Z] WARN  BROKER: Request 'orders.create' is timed out. { requestID: '6cd10b50-…', nodeID: 'app-1', timeout: 1500 }
  ✗ orders.create timeout=1500 → RequestTimeoutError (504) @app-1: Request is timed out when call 'orders.create' action on 'app-1' node. (1506 ms)
--- what app-1 (the orders node) logged meanwhile:
[21:05:23.513Z] WARN  BROKER: Request 'shipping.quote' is timed out. { requestID: '6cd10b50-…', nodeID: 'app-1', timeout: 694.221156 }

Look at the second log line, from the orders node. The shipping.quote call was made with a timeout of 694.22 ms — nobody wrote that number. The client's 1500 ms was carried in the request to app-1; app-1 spent ~806 ms on the inventory hop; when it made the next call, the broker subtracted the elapsed time and passed the remainder down. Both sides timed out at the same millisecond (23.513Z), with the same requestID, and the client got its answer at 1506 ms, not at 1600 or 2300.

In distributed-systems vocabulary this is deadline propagation — the thing gRPC does with deadlines and that most HTTP stacks do not do at all. Without it, a 1.5 s timeout at the edge and a 1.5 s timeout on each inner hop means a request can keep a chain of services busy for 3 s after the caller has already given up on it. With it, the whole tree is cut at the same instant. If the budget is already exhausted when a call is about to be made, the broker does not even send it — you get a RequestSkippedError instead, and that one is deliberately not retryable: there is no budget left to retry with.

5. Fallback: degrade instead of failing

Retry and breaker decide whether to keep trying. When they give up, fallbackResponse decides what the caller gets instead of an exception. Inventory is fully down (FAIL_RATE=1, single node):

const res = await broker.call("inventory.check", { sku: "SKU-1" }, {
  fallbackResponse: (ctx, err) => {
    console.log(`  ↩ fallback used because: ${err.name} (${err.code})`);
    return { sku: ctx.params.sku, inStock: null, stale: true };
  },
});

[21:05:27.838Z] WARN  BROKER: The 'inventory.check' request is failed. Return fallback response. { requestID: '06fa6dc8-…', err: 'inventory database unavailable' }
  ↩ fallback used because: MoleculerRetryableError (503) @inventory-1
  result: { sku: 'SKU-1', inStock: null, stale: true }

fallbackResponse can be a value or a function of (ctx, err) — the function form is where you read from a cache, return the last known value, or return a "we'll confirm by email" answer. The broker sets ctx.fallbackResult = true so tracing and metrics can tell a real answer from a degraded one. On the callee side there is a mirror of this: an action can declare fallback: "methodName" (or a function) in its schema, which runs when the handler throws — useful for the "serve from cache when the DB is down" case where the service knows its own fallback better than every caller does.

A product page that shows "stock: unknown" is a product page. A product page that shows a 500 is not. The fallback is where the difference is made, and it is one option on the call.

6. Bulkhead: don't let one endpoint sink the ship

The last guard is on the callee. orders.report is expensive (300 ms of pretend work) and declares a bulkhead of two concurrent executions and a queue of three:

report: {
  bulkhead: { enabled: true, concurrency: 2, maxQueueSize: 3 },
  async handler(ctx) {
    await new Promise((r) => setTimeout(r, 300));
    return { ok: true, servedBy: this.broker.nodeID };
  },
},

Ten calls fired at once with Promise.allSettled:

  ✓ #0 ok
  ✓ #1 ok
  ✓ #2 ok
  ✓ #3 ok
  ✓ #4 ok
  ✗ #5 QueueIsFullError (429) @app-1: Queue is full. Request 'orders.report' action on 'client-3475145' node is rejected.
  ✗ #6 QueueIsFullError (429) @app-1: Queue is full. …
  ✗ #7 QueueIsFullError (429) @app-1: Queue is full. …
  ✗ #8 QueueIsFullError (429) @app-1: Queue is full. …
  ✗ #9 QueueIsFullError (429) @app-1: Queue is full. …
  total 951 ms

Two run, three wait, five are rejected immediately with a 429 — they do not sit in an unbounded queue eating memory and dragging the p99 of everything else on that node. The five accepted ones finish in three batches (2 + 2 + 1 ≈ 950 ms). The name comes from the watertight compartments in a ship's hull: a flood in one does not sink the whole thing. The bulkhead can be declared per action, as here, or for the whole broker (bulkhead: { enabled: true, concurrency: 10, maxQueueSize: 100 }). Note that QueueIsFullError is marked retryable: with a retry policy on the caller, a rejected call is retried after the backoff, and with several instances of the service that usually lands on a less busy one — a reasonable default. With a single instance it means retrying against the same full queue, which is one more place where a per-action retryPolicy: { enabled: false } earns its keep.

(One quirk visible in the log: the node named in the error message is the caller, because ctx.nodeID on the callee side refers to the sender of the request. The @app-1 prefix — which is err.nodeID, the node that threw — is the one that tells you where the queue actually lives.)

What to put in production

A reasonable baseline that I would start any Moleculer 0.15 project from:

module.exports = {
  requestTimeout: 5 * 1000,
  retryPolicy: { enabled: true, retries: 3, delay: 100, maxDelay: 2000, factor: 2 },
  circuitBreaker: { enabled: true, threshold: 0.5, minRequestCount: 20, windowTime: 60, halfOpenTime: 10 * 1000 },
  bulkhead: { enabled: true, concurrency: 50, maxQueueSize: 200 },
};

Then, per action, turn retries off on anything non-idempotent, set a tighter timeout on anything user-facing, and add fallbacks where a degraded answer is better than none. The defaults you are overriding — no timeout, no retry, no breaker — are the right defaults for a framework that does not know your system, and the wrong ones for any system that has users.

The framework's contribution is not that it has a circuit breaker; every resilience library has one. It is that the breaker, the retry, the timeout, the load balancer and the registry are the same component, so the timeout knows how much budget is left, the retry knows there is another node, and the breaker can take an endpoint out of rotation instead of throwing faster. Three guards, one config block, and the failover in section 3 falls out of it.

Code for this post: examples/06-resilience — run.sh reproduces all six demos against a local NATS. Moleculer 0.15.2, Node 22.

Previous in the series: Service discovery and load balancing in Node.js. Next: event-driven architecture — pub/sub with NATS, Kafka and Redis.

Service discovery and load balancing in Node.js — without Consul or Kubernetes

Icebob — Tue, 29 Sep 2026 13:00:21 +0000

The moment you run a service on more than one machine, two questions appear that a monolith never had to answer: where is it (service discovery) and which copy do I call (load balancing). The standard answers are all infrastructure. A registry like Consul or etcd plus a client library in every service. Kubernetes Service objects plus kube-proxy. A service mesh sidecar next to every pod. An nginx or HAProxy in front of every pool.

Each of these works. Each is also a component you run, monitor, upgrade and get paged for, and none of them knows anything about your application — they see IP addresses and ports, not "the orders.create action, version 2, currently on three nodes".

This article shows the other approach: put the registry in the framework. Every Node.js process keeps a full copy of the service catalogue, learns about changes directly from its peers, and picks the target of each call locally. No registry server, no sidecar, no extra hop. I'll use Moleculer, which has done it this way since 2017, and I'll run every scenario for real — including the one where a node is killed with SIGKILL mid-traffic — so you can see exactly what happens and when.

How it works, in one paragraph

Each process runs a service broker (the Moleculer runtime) with a node ID. When a broker starts, it publishes an INFO packet on the transporter (the message bus — NATS here, but Redis, MQTT, AMQP, Kafka or plain TCP work the same) describing every service, action and event it hosts, plus free-form metadata. Every other broker receives it and updates its local registry. Brokers send a heartbeat every few seconds; a node that stops heartbeating is marked unavailable, and a node that shuts down cleanly says goodbye with a DISCONNECT packet so nobody waits for a timeout. When your code calls broker.call("greeter.hello", …), the caller's registry picks one live endpoint using a strategy and sends the request straight to that node. That is client-side discovery plus client-side load balancing, with the catalogue kept eventually consistent by gossip over the bus.

The setup

One service, deliberately trivial, that reports which node answered:

// services/greeter.service.js — the only service in this article.
// Every response says which node served it, so load balancing is visible.
module.exports = {
  name: "greeter",
  actions: {
    hello: {
      params: { name: "string" },
      handler(ctx) {
        return { hello: ctx.params.name, servedBy: this.broker.nodeID };
      },
    },
  },
};

One config, shared by every worker process and by the client. Everything the article varies is an environment variable; the service never changes:

// moleculer.config.js
const PreferZoneStrategy = require("./prefer-zone.strategy");   // custom strategy, shown later

// Built-in strategies are picked by name; a custom one is passed as a class.
const strategy = process.env.STRATEGY === "PreferZone" ? PreferZoneStrategy : process.env.STRATEGY || "RoundRobin";

module.exports = {
  nodeID: `worker-${process.pid}`,
  transporter: process.env.TRANSPORTER || "nats://localhost:4222",
  metadata: { zone: process.env.ZONE || "eu" }, // free-form node metadata, visible to every other node
  registry: {
    strategy,
    strategyOptions: { shardKey: "name" }, // only Shard reads this: route by ctx.params.name
    preferLocal: true,
  },
  heartbeatInterval: 2, // seconds; default 10
  heartbeatTimeout: 6,  // seconds; default 30 — shortened so the crash demo fits in a terminal
};

Workers are started with moleculer-runner, the framework's CLI host: SERVICEDIR=services npx moleculer-runner --config moleculer.config.js. The client is a broker with no services, which is a perfectly normal thing to be — an API gateway or a cron job looks exactly like this:

// client.js (abridged) — a node with no services of its own; it only calls greeter.hello.
const broker = new ServiceBroker({ ...config, nodeID: `client-${process.pid}` });
await broker.start();
await broker.waitForServices("greeter");            // block until at least one instance is known

for (const name of ["ada", "linus", "grace", "dennis", "ken", "brian"]) {
  const res = await broker.call("greeter.hello", { name });
  console.log(`${name.padEnd(7)} → ${res.servedBy}`);
}

1. Three workers, zero configuration

Start three workers and run the client:

$ node client.js calls 6
ada     → worker-3473401
linus   → worker-3473403
grace   → worker-3473402
dennis  → worker-3473401
ken     → worker-3473403
brian   → worker-3473402

Round-robin across three processes. Nobody registered anything anywhere; the workers announced themselves on the bus and the client's registry did the rest. You can ask any node what it currently knows — the registry is exposed as a built-in $node service:

const nodes = await broker.call("$node.list", { withServices: true });

$ node client.js nodes
client-3473436     available=true  zone=eu  services=-
worker-3473402     available=true  zone=eu  services=greeter
worker-3473401     available=true  zone=eu  services=greeter
worker-3473403     available=true  zone=eu  services=greeter

Note the zone=eu column: that's the metadata from the config, carried in the INFO packet. It will matter in a minute.

2. Scaling down, crashing, scaling up — with traffic running

This is the part registry vendors don't put in the quick-start. The client calls greeter.hello every 500 ms for 16 seconds. While it runs, I stop one worker cleanly (SIGTERM), kill another (SIGKILL, no goodbye), and start a new one.

$ node client.js loop 16
t=  0.0s  → worker-3473034
t=  0.5s  → worker-3473036
t=  1.0s  → worker-3473035
t=  1.5s  → worker-3473034          ← SIGTERM sent to worker-3473034 right after this
t=  2.1s  → worker-3473036
t=  2.6s  → worker-3473036
t=  3.1s  → worker-3473035
t=  3.6s  → worker-3473036
t=  4.1s  → worker-3473035
t=  4.6s  → worker-3473036
t=  5.1s  → worker-3473035
t=  5.6s  → worker-3473036          ← SIGKILL sent to worker-3473035 here
[20:45:57.784Z] WARN  BROKER: Request 'greeter.hello' is timed out. { nodeID: 'worker-3473035', timeout: 2000 }
t=  6.2s  ✗ RequestTimeoutError: Request is timed out when call 'greeter.hello' action on 'worker-3473035' node.
t=  8.7s  → worker-3473036
[20:46:00.807Z] WARN  BROKER: Request 'greeter.hello' is timed out. { nodeID: 'worker-3473035', timeout: 2000 }
t=  9.2s  ✗ RequestTimeoutError: Request is timed out when call 'greeter.hello' action on 'worker-3473035' node.
t= 11.7s  → worker-3473036
[20:46:01.618Z] WARN  DISCOVERY: Heartbeat is not received from 'worker-3473035' node.
[20:46:01.619Z] WARN  REGISTRY: Node 'worker-3473035' disconnected unexpectedly.
t= 12.2s  → worker-3473036
t= 12.7s  → worker-3473036
t= 13.3s  → worker-3473036
t= 13.8s  → worker-3473098          ← new worker, first request within a second of starting
t= 14.3s  → worker-3473036
t= 14.8s  → worker-3473098

Three different things happened, and it's worth being precise about each:

Graceful stop is instant. worker-3473034 got SIGTERM, the runner stopped the broker, the broker sent DISCONNECT, and the node was gone from every registry before the next call. Zero failed requests. This is what happens on every deploy, so it's the case that matters most.
A crash is detected by heartbeat, and until then calls to that node fail. worker-3473035 was killed at ~5.6 s. The registry only learned at ~11.6 s (heartbeatTimeout: 6), and in between, every request round-robined onto the dead node timed out. That's two failures out of eight calls. With the default heartbeatTimeout: 30 it would have been worse. No client-side registry can do better than its heartbeat window — Consul's agent has the same gap, Kubernetes' readiness probes have the same gap — the difference is only who notices and how you react. Reacting is what retries and circuit breakers are for; that's the next article. For now: tune heartbeatInterval/heartbeatTimeout to what your deployment can afford (2/6 is fine for a dozen nodes; the defaults are conservative for hundreds).
Scaling up is sub-second. worker-3473098 started, published INFO, and was serving traffic on the next tick. No registration step, no health-check grace period, no DNS TTL.

3. Choosing who gets the call: strategies

Round-robin is the default, but the strategy is a one-word option. Built in:

RoundRobin — the default; equal share, in order.
Random — equal share, no order; useful when many callers would otherwise stay in lockstep.
CpuUsage — samples the CPU load nodes report in their heartbeat, picks a lightly loaded one. Good for uneven hardware.
Latency — measures round-trip time to each node and prefers the fast ones. Good for multi-region.
Shard — consistent hashing on a field from the request, so the same key always lands on the same node. This is how you get in-memory per-user caches or ordered processing per key without a separate sticky-session layer.

Shard, keyed on name, same three workers:

$ STRATEGY=Shard node client.js calls 12
ada     → worker-3473403
linus   → worker-3473403
grace   → worker-3473471
dennis  → worker-3473471
ken     → worker-3473471
brian   → worker-3473471
ada     → worker-3473403
linus   → worker-3473403
grace   → worker-3473471
dennis  → worker-3473471
ken     → worker-3473471
brian   → worker-3473471

Same name, same node, every time — with a consistent-hash ring, so adding a node only moves the keys that land on it. Two things I learned while writing this: the key must be given as registry.strategyOptions.shardKey (I first tried strategy: { type: "Shard", options: {…} }, which is silently ignored and falls back to random selection — check the output, not the config), and the key can be read from ctx.meta instead of params with a # prefix ("#userId"), which is usually what you want since the user ID rides in meta. Strategies can also be set per action (strategy/strategyOptions on the action definition) if one hot action needs sharding and the rest don't.

Writing your own in fifteen lines

The strategy interface is one method: select(endpoints, ctx). Here's one that keeps traffic inside the caller's own zone when it can, and falls back to everything when it can't — the classic "same-AZ first" rule, without a mesh:

// prefer-zone.strategy.js — a custom load-balancing strategy in ~15 lines.
// Picks an instance in the caller's own zone when one is alive; falls back to any instance.
const { Strategies } = require("moleculer");

class PreferZoneStrategy extends Strategies.Base {
  constructor(registry, broker, opts) {
    super(registry, broker, opts);
    this.zone = broker.metadata.zone;
    this.fallback = new Strategies.RoundRobin(registry, broker, opts);
  }

  select(endpoints, ctx) {
    const local = endpoints.filter((ep) => ep.node.metadata?.zone === this.zone);
    return this.fallback.select(local.length ? local : endpoints, ctx);
  }
}

module.exports = PreferZoneStrategy;

Add a fourth worker with ZONE=us, then call from three different places:

$ node client.js nodes
worker-3473488     available=true  zone=us  services=greeter
worker-3473471     available=true  zone=eu  services=greeter
worker-3473403     available=true  zone=eu  services=greeter

$ STRATEGY=PreferZone ZONE=eu node client.js calls 6
ada     → worker-3473471
linus   → worker-3473403
grace   → worker-3473471
dennis  → worker-3473403
ken     → worker-3473471
brian   → worker-3473403

$ STRATEGY=PreferZone ZONE=us node client.js calls 6
ada     → worker-3473488
linus   → worker-3473488
grace   → worker-3473488
dennis  → worker-3473488
ken     → worker-3473488
brian   → worker-3473488

$ STRATEGY=PreferZone ZONE=asia node client.js calls 6      # no local instance → fall back to all
ada     → worker-3473403
linus   → worker-3473471
grace   → worker-3473488
dennis  → worker-3473403
ken     → worker-3473471
brian   → worker-3473488

The eu caller round-robins over the two eu workers, the us caller pins to its one us worker, and a caller from a zone with no workers uses all three. The metadata came from the registry — the strategy never made a network call. You can put anything in metadata: version, hardware class, canary flag, tenant — and route on it the same way.

One more rule you get for free: `preferLocal`

If the calling node itself hosts an instance of the service, preferLocal: true (the default) calls it in-process and skips the network entirely. That's what lets the same code run as a modular monolith on a laptop and as a distributed system in production — I showed that end-to-end in the previous article.

4. No message broker either: discovery over plain TCP

Everything above used NATS as the bus. If you don't want to run even that — a small deployment, an on-prem box, an edge device — Moleculer's TCP transporter has discovery built in: nodes find each other by UDP multicast and then talk over direct TCP connections. Same two workers, same client, one variable changed:

$ TRANSPORTER=TCP SERVICEDIR=services npx moleculer-runner --config moleculer.config.js
[20:47:29.966Z] INFO  TRANSPORTER: TCP server is listening on port 42799
[20:47:29.971Z] INFO  TRANSPORTER: UDP Multicast Server is listening on 192.168.1.20:4445. Membership: 239.0.0.0
[20:47:29.972Z] INFO  TRANSPORTER: UDP discovery started.

$ TRANSPORTER=TCP node client.js calls 4
ada     → worker-3473535
linus   → worker-3473536
grace   → worker-3473535
dennis  → worker-3473536

(IP address replaced; everything else is the actual log.) Service discovery and load balancing with no infrastructure process at all — not a registry, not a bus. The honest limitation: most cloud networks block multicast between VMs, so across hosts on AWS/GCP/Azure you either give the TCP transporter a static urls list of peers, or you use NATS/Redis, which is what I'd do anyway once there's more than one host.

When you still want Consul or Kubernetes

Being fair about the boundaries:

Polyglot systems. The registry lives in the Node.js processes. A Go or Python service can't join it (there are Java and Go ports that speak the same protocol, but they're separate community projects). If half your services aren't Node, an external registry is the lingua franca.
You're already on Kubernetes. Then keep it — for scheduling, restarts, secrets and ingress, it's excellent. It doesn't conflict: Kubernetes decides which pods exist, the broker's registry decides which action call goes to which pod and does so with application-level knowledge (versions, metadata, per-action strategies) that a Service object doesn't have. Many teams run exactly this combination, and it means the same code also runs on a plain VM.
Very large clusters. Every node hearing every heartbeat is O(n²) chatter. Past a few hundred nodes, switch the discoverer from the default gossip to the Redis- or etcd-backed one (registry.discoverer: "Redis") — heartbeats go to a shared store instead of the bus. That's one config line, and the rest of the article is unchanged.

For a Node.js system of, say, five to fifty services on a handful of machines — which is most systems — the registry in the framework covers service discovery, load balancing, zone affinity, sharding and node health, and the only infrastructure you ran for it in this article was one NATS container. Or, in the last demo, nothing.

Code in this article was run on Moleculer 0.15.2 and Node.js 22, with NATS 2 as the bus except in the TCP section. The moleculer-examples repository has a run.sh that reproduces all five demos, including the crash-and-recover timeline.

From a modular monolith to microservices without a rewrite

Icebob — Tue, 22 Sep 2026 13:00:14 +0000

Most "monolith to microservices" stories go one of two ways. Either it's a rewrite that takes eighteen months and ships a system nobody asked for, or it's a big-bang split into fifteen services that immediately need a Kubernetes team. Both come from the same mistake: treating "process boundaries" and "module boundaries" as the same decision.

They aren't. You can draw the module boundaries now, inside the monolith, and move the process boundaries later — one service at a time, when you have a reason to. This article shows exactly that with a small Express app: three stages, the same service files in every stage, and real output at each step, including the one thing that does break when you finally run two copies of a service.

The starting point

A perfectly ordinary Express monolith. Three modules, wired together with require(), one process.

// lib/orders.js — calls users and mailer through plain imports. Tight coupling, zero ceremony.
const users = require("./users");
const mailer = require("./mailer");

let counter = 0;
const orders = [];

exports.create = ({ userId, item, amount }) => {
  const user = users.get(userId);
  const order = { id: ++counter, userId, item, amount };
  orders.push(order);
  mailer.send(user.email, `Order #${order.id} confirmed (${item})`);
  return order;
};

// app.js
const express = require("express");
const users = require("./lib/users");
const orders = require("./lib/orders");

const app = express();
app.use(express.json());

app.get("/users/:id", (req, res) => res.json(users.get(Number(req.params.id))));
app.post("/orders", (req, res) => {
  const userId = Number(req.header("x-user-id"));
  res.status(201).json(orders.create({ userId, ...req.body }));
});

app.listen(process.env.PORT || 3000);

$ curl -s localhost:3000/users/1
{"id":1,"name":"Ada","email":"ada@example.com"}
$ curl -s -X POST localhost:3000/orders -H 'content-type: application/json' -H 'x-user-id: 2' -d '{"item":"Monitor","amount":329}'
{"id":1,"userId":2,"item":"Monitor","amount":329}

Nothing wrong with this. It's the right architecture for a team of three with one deployable. The problem only starts when orders grows a queue consumer, mailer needs to scale independently because a marketing campaign sends 200k emails, and you realise nothing in the codebase says which module is allowed to call which.

Stage 1: module boundaries become service boundaries — same process

Bring in a service broker (Moleculer's ServiceBroker — think of it as an in-process service runtime with a registry) and turn each module into a service: a plain object with a name and actions (its callable endpoints). No transporter is configured, so every call is an in-memory function call. No network, no serialization, no new infrastructure.

// services/orders.service.js — lib/orders.js as a service.
// No require("./users") any more: the dependency goes through the broker, and is declared.
let counter = 0;                       // module-level state — fine in a monolith; see "What breaks" below
const orders = [];

module.exports = {
  name: "orders",
  dependencies: ["users"],             // broker waits for `users` before starting this service
  actions: {
    create: {
      params: {
        userId: { type: "number", convert: true },
        item: "string",
        amount: { type: "number", positive: true },
      },
      async handler(ctx) {
        const user = await ctx.call("users.get", { id: ctx.params.userId });
        const order = { id: ++counter, userId: user.id, item: ctx.params.item, amount: ctx.params.amount };
        orders.push(order);
        await ctx.emit("order.created", { order, user });
        return { ...order, servedBy: this.broker.nodeID, usersServedBy: user.servedBy };
      },
    },
  },
};

Three things changed and they are all improvements you'd want anyway:

The dependency is explicit. dependencies: ["users"] is documentation the runtime enforces — orders won't start until users is available.
The input is validated at the boundary. params is a schema; a bad amount is rejected before the handler runs.
Mailer is no longer called — it's notified. orders emits order.created; whoever cares subscribes. That's the seam you'll use later to move it out.

// services/mailer.service.js — lib/mailer.js as a service. Instead of being called, it reacts to an event.
module.exports = {
  name: "mailer",
  events: {
    "order.created"(ctx) {
      const { order, user } = ctx.params;
      this.logger.info(`→ ${user.email}: Order #${order.id} confirmed (${order.item})`);
    },
  },
};

Express stays. The routes call the broker instead of the modules:

// app.js — the same Express app, now with a broker inside. Routes call services instead of modules.
const express = require("express");
const { ServiceBroker } = require("moleculer");

const broker = new ServiceBroker({
  nodeID: `app-${process.pid}`,
  transporter: process.env.TRANSPORTER || null,     // null = local bus, in-process only
  logger: { type: "Console", options: { level: "info", formatter: "short" } },
});

const wanted = process.env.SERVICES === undefined ? ["users", "orders", "mailer"] : process.env.SERVICES.split(",").filter(Boolean);
for (const name of wanted) broker.createService(require(`./services/${name}.service.js`));

const app = express();
app.use(express.json());

app.get("/users/:id", async (req, res, next) => {
  try { res.json(await broker.call("users.get", { id: req.params.id })); } catch (e) { next(e); }
});
app.post("/orders", async (req, res, next) => {
  try {
    const userId = Number(req.header("x-user-id"));
    res.status(201).json(await broker.call("orders.create", { userId, ...req.body }));
  } catch (e) { next(e); }
});
app.use((err, req, res, next) => res.status(err.code || 500).json({ error: err.name, message: err.message }));

broker.start().then(() => {
  app.listen(process.env.PORT || 3000, () => broker.logger.info(`HTTP on ${process.env.PORT || 3000}`));
});

$ node app.js
[20:06:31.508Z] INFO  BROKER: Node ID: app-3470689
[20:06:31.549Z] INFO  ORDERS: Waiting for service(s) 'users'...
[20:06:31.557Z] INFO  USERS: Service 'users' started.
[20:06:31.557Z] INFO  MAILER: Service 'mailer' started.
[20:06:32.552Z] INFO  ORDERS: Service(s) 'users' are available.
[20:06:32.554Z] INFO  ORDERS: Service 'orders' started.
[20:06:32.554Z] INFO  BROKER: ✔ ServiceBroker with 4 service(s) started successfully in 1s.
[20:06:32.557Z] INFO  BROKER: HTTP on 3000

$ curl -s localhost:3000/users/1
{"id":1,"name":"Ada","email":"ada@example.com","servedBy":"app-3470689"}
$ curl -s -X POST localhost:3000/orders -H 'content-type: application/json' -H 'x-user-id: 2' -d '{"item":"Monitor","amount":329}'
{"id":1,"userId":2,"item":"Monitor","amount":329,"servedBy":"app-3470689","usersServedBy":"app-3470689"}
# app log:
[20:06:34.088Z] INFO  MAILER: → linus@example.com: Order #1 confirmed (Monitor)

Same behaviour, same single process, same deploy. servedBy and usersServedBy are the same node because everything is local. You could stop here for a year and still be better off than before: the boundaries are real, the contracts are validated, and nobody can sneak a require("../orders/db") across modules any more.

This is what "modular monolith" should mean in practice — not a folder convention, but boundaries a runtime knows about.

Stage 2: extract one service — the one that actually needs it

Marketing week: mailer needs to scale on its own and must not slow down the request path. So move only mailer out.

Two changes, neither of them in service code. First, give the app a transporter (a message broker — NATS here) and tell it which services to host locally:

$ TRANSPORTER=nats://localhost:4222 SERVICES=users,orders node app.js

Second, start mailer in its own process with moleculer-runner, the framework's CLI service host, pointed at the same services/ folder:

// moleculer.config.js — for services that run OUTSIDE the app process, via moleculer-runner.
module.exports = {
  nodeID: `svc-${process.env.SERVICES || "all"}-${process.pid}`,
  transporter: process.env.TRANSPORTER || "nats://localhost:4222",
  logger: { type: "Console", options: { level: "info", formatter: "short" } },
};

$ SERVICEDIR=services SERVICES=mailer npx moleculer-runner --config moleculer.config.js

Now hit the same endpoints:

$ curl -s -X POST localhost:3000/orders -H 'content-type: application/json' -H 'x-user-id: 2' -d '{"item":"Monitor","amount":329}'
{"id":1,"userId":2,"item":"Monitor","amount":329,"servedBy":"app-3470296","usersServedBy":"app-3470296"}

# mailer process log:
[20:03:06.188Z] INFO  MAILER: → linus@example.com: Order #1 confirmed (Monitor)

Look at what happened:

mailer received the order.created event in another process, over NATS. The orders code that emits it didn't change.
orders → users is still app-3470296 → app-3470296: in-process. The app is connected to NATS, but the registry's default preferLocal: true routes a call to a local instance whenever one exists. You extracted one service and paid the network cost for exactly one hop — the one you chose.

That's the whole method. The services/ folder is the same in both processes; which process loads which file is a deployment decision, made per environment with two environment variables. Your dev laptop runs node app.js with everything local; staging runs the hybrid; production splits further. Same code.

Stage 3: split the rest, scale one — and meet the thing that breaks

Let's go all the way: users, orders and mailer each in their own process, two instances of orders because it's the hot path, and the app hosting nothing but HTTP:

$ SERVICEDIR=services SERVICES=users  npx moleculer-runner --config moleculer.config.js
$ SERVICEDIR=services SERVICES=orders npx moleculer-runner --config moleculer.config.js
$ SERVICEDIR=services SERVICES=orders npx moleculer-runner --config moleculer.config.js
$ SERVICEDIR=services SERVICES=mailer npx moleculer-runner --config moleculer.config.js
$ TRANSPORTER=nats://localhost:4222 SERVICES= node app.js

$ for i in 1 2 3 4; do curl -s -X POST localhost:3000/orders -H 'content-type: application/json' -H 'x-user-id: 1' -d '{"item":"Cable","amount":9}'; echo; done
{"id":1,"userId":1,"item":"Cable","amount":9,"servedBy":"svc-orders-3470323","usersServedBy":"svc-users-3470321"}
{"id":1,"userId":1,"item":"Cable","amount":9,"servedBy":"svc-orders-3470322","usersServedBy":"svc-users-3470321"}
{"id":2,"userId":1,"item":"Cable","amount":9,"servedBy":"svc-orders-3470323","usersServedBy":"svc-users-3470321"}
{"id":2,"userId":1,"item":"Cable","amount":9,"servedBy":"svc-orders-3470322","usersServedBy":"svc-users-3470321"}

The good news: round-robin load balancing between the two orders instances, calls to users over the wire, no code changes — all of it just works.

The bad news is in the first column. Order IDs 1, 1, 2, 2. Remember let counter = 0 at the top of orders.service.js? Each process has its own. In the monolith it was a perfectly good ID generator; with two instances it's a duplicate-key bug that would have shipped.

This is the honest part of the article. Turning modules into services is nearly free. Running two copies of one is where the monolith's hidden assumptions surface. Here is the list I check before scaling any service past one instance:

Module-level state — counters, caches, "the current batch", Maps of sessions. Every instance gets its own. Move it to the database or a shared store, or make it instance-safe (UUIDs instead of counters; here id: crypto.randomUUID() is the one-line fix).
In-memory cache. Moleculer's default Memory cacher is per node; with several instances you'll serve stale data from one and fresh from another. Switch the cacher to Redis — a broker option, not a code change.
Anything that assumes ordering across requests. Two instances process concurrently; if order mattered, you were relying on a single event loop.
Transactions across services. orders + inventory in one SQL transaction worked because they were one process on one connection. Across processes it's either a saga (compensating actions on failure) or you keep those two in the same service. Both are fine; pretending it's still atomic is not.
ctx.meta instead of globals. The request-scoped things you used to keep in req or a module variable (user ID, locale, trace ID) now need to ride along explicitly. Moleculer propagates ctx.meta through every call and event automatically — put them there.

None of these are Moleculer problems, and none of them are avoided by any other framework; they're the actual difference between one process and two. The point of the staged approach is that you hit them one service at a time, with a working system on both sides, instead of all at once on cut-over day.

The migration order that works

If I were doing this to a real codebase:

Stage 1 for everything, no exceptions. Broker in the monolith, every module a service, every cross-module call a ctx.call, every fire-and-forget a ctx.emit. Ship it. It is a refactor with no infrastructure change, and it's where you discover which modules actually depend on which.
Extract the asynchronous ones first. Mailer, image processing, report generation, webhooks — anything that already receives events rather than answering calls. They have no callers to break and their latency doesn't sit on the request path.
Extract what needs to scale or deploy independently. Usually one or two hot services. Run through the "what breaks" list before starting the second instance.
Leave the rest together. A modular monolith with two extracted services is a legitimate end state, not a half-finished migration. Every boundary you didn't turn into a network hop is one that can't fail at 3 a.m.

And keep node app.js — the everything-local mode — working forever. It's how a new developer runs the whole system on a laptop with no Docker, and it's how you run integration tests without a message broker. The local bus is a feature, not a stepping stone.

Code in this article was run on Moleculer 0.15.2, Express 5 and Node.js 22, with NATS 2 for stages 2 and 3. The moleculer-examples repository has a run.sh that reproduces all three stages — including the duplicate-ID bug — in one go.

Using Moleculer with TypeScript: contracts, two service styles, and the honest caveats

Icebob — Tue, 15 Sep 2026 13:00:42 +0000

"Does Moleculer work with TypeScript?" is the second most common question I get about it, right after the NestJS one. The short answer is yes — it ships its own type definitions, you can write services as typed schemas or as classes, and the runner loads .ts files. The longer answer has a few honest caveats, and since you'd hit them on day two anyway, I'd rather put them in the article.

This post shows a small typed system end to end: a shared contract file, two services in the two supported styles, a caller that gets autocomplete and compile errors for wrong action names, and the dev and production ways to run it. Everything below was compiled with strict: true and executed as shown, on Moleculer 0.15.2, Node.js 22 and TypeScript 7.

What Moleculer gives you out of the box

Bundled typings. moleculer ships index.d.ts; there is no @types/moleculer. The generics that matter: ServiceSchema<TSettings, TMethods, TVars> for typing this inside a service, Context<TParams, TMeta> for typing what a handler receives, and broker.call<TReturn, TParams>() for typing a call.
Class-based services. class X extends Service with this.parseServiceSchema({...}) in the constructor is a first-class way to define a service, not a hack.
.ts loading in the runner. moleculer-runner resolves moleculer.config.ts and *.service.ts files as long as a TypeScript loader is registered in the process (tsx, ts-node).
A TypeScript project template for the CLI (moleculer-cli → moleculer init project-typescript my-app), maintained alongside the framework.

What it does not give you — and I'll come back to each — is a decorator API in core, dependency injection, or any type information that crosses the network. That last one is not a Moleculer limitation, it is how distributed systems work, and the trick below is how you get most of the benefit anyway.

Step 1: a contract file

Action names are strings, and by default broker.call("products.get", { id: 1 }) returns Promise<any>. The fix is one file that says what each action takes and returns. Moleculer doesn't generate it; you write it, and both the service and every caller import it.

// src/contracts.ts — shared types for the actions in this system.
export interface Product { id: number; name: string; price: number; }
export interface Order { id: string; productId: number; quantity: number; total: number; }

/** Action name → { params, result }. */
export interface Actions {
  "products.get":  { params: { id: number };                       result: Product };
  "products.list": { params: { limit?: number };                   result: Product[] };
  "orders.create": { params: { productId: number; quantity: number }; result: Order };
}

/** Events and their payloads. */
export interface Events {
  "order.created": Order;
}

/** What every request carries in ctx.meta. */
export interface Meta { userId?: string; }

Then a ~25-line wrapper turns broker.call and ctx.emit into typed functions:

// src/typed-broker.ts
import type { Context, ServiceBroker, CallingOptions } from "moleculer";
import type { Actions, Events, Meta } from "./contracts";

export type ActionName = keyof Actions;
export type ActionParams<N extends ActionName> = Actions[N]["params"];
export type ActionResult<N extends ActionName> = Actions[N]["result"];

/** A Context whose params/meta are the ones declared for `N`. */
export type Ctx<N extends ActionName> = Context<ActionParams<N>, Meta>;
/** Same for event handlers: ctx.params is the event payload. */
export type EventCtx<E extends keyof Events> = Context<Events[E], Meta>;

export function call<N extends ActionName>(
  caller: ServiceBroker | Context<unknown, Meta>,
  action: N,
  params: ActionParams<N>,
  opts?: CallingOptions,
): Promise<ActionResult<N>> {
  return caller.call<ActionResult<N>, ActionParams<N>>(action, params, opts);
}

export function emit<E extends keyof Events>(ctx: Context<unknown, Meta>, event: E, payload: Events[E]) {
  return ctx.emit(event, payload);
}

This is the whole trick. Everything else in the article builds on it.

Step 2: a service in schema style

The schema object is Moleculer's native idiom. In TypeScript you annotate it as ServiceSchema<Settings, Methods>, and this.settings.currency and this.findOrFail() become type-checked inside handlers.

// services/products.service.ts
import type { ServiceSchema, Context } from "moleculer";
import type { Product, Meta } from "../src/contracts";

interface Settings { currency: string; }
interface Methods { findOrFail(id: number): Product; }

const catalogue: Product[] = [
  { id: 1, name: "Keyboard", price: 79 },
  { id: 2, name: "Monitor", price: 329 },
  { id: 3, name: "Cable", price: 9 },
];

const ProductsService: ServiceSchema<Settings, Methods> = {
  name: "products",
  settings: { currency: "EUR" },

  actions: {
    get: {
      // Runtime validation — the compiler cannot check what arrives over the wire.
      params: { id: { type: "number", convert: true } },
      handler(ctx: Context<{ id: number }, Meta>): Product {
        this.logger.info(`get #${ctx.params.id} for user ${ctx.meta.userId ?? "anonymous"}`);
        return this.findOrFail(ctx.params.id);          // typed via Methods
      },
    },
    list: {
      params: { limit: { type: "number", optional: true, convert: true } },
      handler(ctx: Context<{ limit?: number }, Meta>): Product[] {
        return catalogue.slice(0, ctx.params.limit ?? catalogue.length);
      },
    },
  },

  methods: {
    findOrFail(id) {
      const p = catalogue.find(p => p.id === id);
      if (!p) throw new Error(`Product ${id} not found`);
      return p;
    },
  },

  started() {
    this.logger.info(`products up, prices in ${this.settings.currency}`);   // typed via Settings
  },
};

export default ProductsService;

Two things to notice. The params block is runtime validation (Moleculer uses fastest-validator) and it stays there no matter how much TypeScript you add — the compiler has no idea what another process will send. And the Context<{ id: number }, Meta> annotation on the handler is what makes ctx.params.id a number rather than unknown; you could use the Ctx<"products.get"> alias from the wrapper instead, which is what the next service does.

Step 3: a service in class style

If your team thinks in classes, extend Service and pass the schema from the constructor. Handlers are plain methods, this is the class, and private state is a field.

// services/orders.service.ts
import { Service, ServiceBroker } from "moleculer";
import type { Order } from "../src/contracts";
import { call, emit, type Ctx, type EventCtx } from "../src/typed-broker";

export default class OrdersService extends Service {
  private orders: Order[] = [];

  constructor(broker: ServiceBroker) {
    super(broker);
    this.parseServiceSchema({
      name: "orders",
      actions: {
        create: {
          params: {
            productId: { type: "number", convert: true },
            quantity: { type: "number", min: 1, convert: true },
          },
          handler: this.create,
        },
      },
      events: {
        "order.created": this.onOrderCreated,
      },
    });
  }

  async create(ctx: Ctx<"orders.create">): Promise<Order> {
    // Typed call: the action name autocompletes, params are checked, the result is a Product.
    const product = await call(ctx, "products.get", { id: ctx.params.productId });

    const order: Order = {
      id: `ord_${this.orders.length + 1}`,
      productId: product.id,
      quantity: ctx.params.quantity,
      total: product.price * ctx.params.quantity,
    };
    this.orders.push(order);

    await emit(ctx, "order.created", order);   // payload must be an Order
    return order;
  }

  onOrderCreated(ctx: EventCtx<"order.created">) {
    this.logger.info(`event: order ${ctx.params.id} created, total ${ctx.params.total}`);
  }
}

Moleculer binds handlers to the service instance, so passing this.create as the handler is fine — inside it, this.orders and this.logger are what you expect.

Step 4: a typed caller

// src/client.ts
import { ServiceBroker } from "moleculer";
import { call } from "./typed-broker";

const broker = new ServiceBroker({
  nodeID: `client-${process.pid}`,
  transporter: { type: "NATS", options: { url: "nats://localhost:4222" } },
  logger: false,
});

async function main() {
  await broker.start();
  await broker.waitForServices(["products", "orders"]);

  const product = await call(broker, "products.get", { id: 2 });
  console.log("product:", product.name, product.price);        // product: Product

  const order = await call(broker, "orders.create", { productId: 2, quantity: 3 });
  console.log("order:", order.id, "total", order.total);       // order: Order

  // Runtime validation still guards the wire: force a bad payload past the compiler.
  try {
    await call(broker, "products.get", { id: "two" as unknown as number });
  } catch (err: any) {
    console.log("rejected:", err.name, "-", err.data?.[0]?.message);
  }

  await broker.stop();
}
main();

And here is what the compiler says when you get it wrong — this is the payoff of the contract file:

await call(broker, "products.get", { id: "2" });     // wrong param type
await call(broker, "product.get", { id: 2 });        // typo in the action name
const order = await call(broker, "orders.create", { productId: 2, quantity: 1 });
console.log(order.totl);                             // typo in a result field

$ npx tsc --noEmit -p .
src/mistakes.ts(8,40): error TS2322: Type 'string' is not assignable to type 'number'.
src/mistakes.ts(9,22): error TS2345: Argument of type '"product.get"' is not assignable to parameter of type 'keyof Actions'.
src/mistakes.ts(11,21): error TS2551: Property 'totl' does not exist on type 'Order'. Did you mean 'total'?

Three classes of bug that used to be runtime failures in another process are now red squiggles in the editor.

Step 5: running it

Development: run the .ts sources directly. moleculer-runner resolves .ts config and service files; it just needs a loader in the process, and tsx is the least fuss:

// moleculer.config.ts
import type { BrokerOptions } from "moleculer";

const config: BrokerOptions = {
  nodeID: `node-${process.pid}`,
  transporter: { type: "NATS", options: { url: process.env.TRANSPORTER ?? "nats://localhost:4222" } },
  logger: { type: "Console", options: { formatter: "short" } },
  requestTimeout: 2000,
};
export default config;

$ node --import tsx node_modules/.bin/moleculer-runner --config moleculer.config.ts --mask "**/*.service.ts" services
[17:49:58.859Z] INFO  PRODUCTS: products up, prices in EUR
[17:49:58.868Z] INFO  REGISTRY: 'orders' service is registered.
[17:49:58.869Z] INFO  REGISTRY: 'products' service is registered.
[17:49:58.870Z] INFO  BROKER: ✔ ServiceBroker with 3 service(s) started successfully in 869ms.

$ npx tsx src/client.ts
product: Monitor 329
order: ord_1 total 987
rejected: ValidationError - The 'id' field must be a number.

The --mask matters: the runner's default file mask is **/*.service.js, so without it your .ts services are silently skipped.

Production: compile once, ship JavaScript, run with plain Node. No loader in the hot path.

$ npx tsc -p .
$ node node_modules/.bin/moleculer-runner --config dist/moleculer.config.js dist/services
$ node dist/src/client.js
product: Monitor 329
order: ord_1 total 987
rejected: ValidationError - The 'id' field must be a number.

Same output, same services. In Docker this is a two-stage build: tsc in the first stage, only dist/ and production dependencies in the second.

The honest caveats

1. No decorators in core. If you want @Action() / @Event() on class methods, the community package moleculer-decorators does it. I tested version 1.3.0 against Moleculer 0.15.2 and it works:

import { ServiceBroker, Context, Service as MoleculerService } from "moleculer";
import { Service, Action, Event, Method } from "moleculer-decorators";

@Service({ name: "greeter", settings: { greeting: "Hello" } })
class GreeterService extends MoleculerService<{ greeting: string }> {
  @Action({ params: { name: "string" } })
  hello(ctx: Context<{ name: string }>) {
    return this.format(ctx.params.name);
  }

  @Method
  format(name: string) {
    return `${this.settings.greeting}, ${name}!`;
  }

  @Event()
  "user.signup"(ctx: Context<{ name: string }>) {
    this.logger.info(`welcome mail to ${ctx.params.name}`);
  }
}

$ npx tsx greeter.service.ts
Hello, Ada!

But be aware of what you're taking on: the package's last release was in 2022, it declares a dependency on Moleculer 0.14, and it uses the legacy experimentalDecorators flag rather than the standard (TC39) decorators that TypeScript 5+ supports. It is small and readable, so forking it is a realistic fallback — but it is not maintained by the core team, and I won't pretend otherwise. If decorators are a hard requirement for your team, weigh that.

2. Types don't cross the wire. The Actions interface is a promise you make to yourself. If the products service is deployed with a new field and the caller isn't, the compiler is happy and the runtime is what it is. Keep the contract file in a shared package (or a monorepo) and version it with the services. Runtime params validation stays mandatory — it is the only thing that checks what actually arrived.

3. Some option types are stricter than the runtime. Example you'll hit in the first five minutes: transporter: "nats://localhost:4222" is the documented shorthand and works at runtime, but BrokerOptions["transporter"] doesn't accept an arbitrary string, so under strict you'll get a type error. Use the object form { type: "NATS", options: { url } }, which is better anyway once you have more than one option. Expect a handful of these; the typings are hand-written and improve release by release (0.15 reworked them — this is typed in lifecycle handlers, for instance), and pull requests to them are the easiest way to contribute to the project.

4. No dependency injection. Moleculer has mixins for shared behaviour and this.broker for reaching other services, not a DI container. If your architecture leans on injectable providers and testing through container overrides, that's a real difference — see the NestJS comparison for when it matters.

5. this typing depends on you. Inside a schema, this is only as typed as the generics you pass to ServiceSchema. Skip them and this.settings.anything is any. It's a small discipline, but it is a discipline.

Recommendations

Start with the contract file + typed wrapper. It's the highest-value 50 lines in a Moleculer TypeScript project and it works with either service style.
Use schema style by default; it matches the docs and every example on the internet. Use class style when a service has enough private state and helpers that a class reads better.
Use Ctx<"action.name"> for handler signatures so the params type is defined once, in the contract, not repeated per handler.
tsx in development, tsc → dist/ in production. Don't ship a TypeScript loader to prod.
Keep strict: true and skipLibCheck: true — the second one saves you from type-checking your entire node_modules on every build.

If you already have a JavaScript Moleculer project, none of this requires a rewrite: rename a file to .ts, add the contract for the actions it calls, and go one service at a time.

Code in this article was compiled with TypeScript 7.0 (strict: true) and run on Moleculer 0.15.2, Node.js 22 and NATS 2. The full example, including a run.sh that exercises both the dev and the production path, is in the moleculer-examples repository.

Moleculer vs NestJS: which one, and when

Icebob — Mon, 07 Sep 2026 19:01:54 +0000

If you are choosing a framework for a Node.js backend that will be more than one service, you will end up comparing NestJS and Moleculer. People ask me about it constantly, usually phrased as "which is better?", and I maintain one of them, so you would expect a sales pitch.

You are not getting one. NestJS and Moleculer solve different problems and overlap in the middle, and the honest answer is that most teams should pick based on which problem they actually have. This article lays out the difference, shows the same service written in both, and ends with a decision list I would actually stand behind.

First, what each one is

NestJS is an application framework. It gives a Node.js codebase structure: modules, dependency injection, decorators, guards, pipes, interceptors, and a CLI that scaffolds all of it. It is TypeScript-first and Angular-shaped, and it has integrations for nearly everything — REST, GraphQL, WebSockets, Swagger, task scheduling, caching, queues. Its @nestjs/microservices package lets a Nest app listen on a message transport (TCP, Redis, NATS, MQTT, RabbitMQ, Kafka, gRPC) instead of HTTP, and call other apps over it.

Moleculer is a microservices framework. Its core concern is the distributed-systems layer: a service registry with discovery, load balancing between instances, request/response and pub/sub messaging over a pluggable transporter, and resilience — timeouts, retries, circuit breaker, bulkhead, fallbacks — built into every call. It also has caching, parameter validation, metrics and distributed tracing as broker options. It is less opinionated about how you structure the code inside a service: a service is a plain object with actions and event handlers.

The overlap is "several Node.js processes talking to each other over NATS". The difference is what each framework takes responsibility for once they do.

The same service, twice

Let's build the smallest realistic thing: a products service that runs as its own process, behind an HTTP gateway, with two instances for redundancy. Both versions use NATS. All of the code below runs as shown — Node 22, NestJS 12, Moleculer 0.15.

NestJS

The service is a controller with a message pattern, bootstrapped as a microservice. The queue option puts every instance in the same NATS queue group, which is what gives you load balancing.

// nest/products.service.ts — a NestJS "microservice": a NATS listener with message patterns.
import "reflect-metadata";
import { Controller, Module } from "@nestjs/common";
import { NestFactory } from "@nestjs/core";
import { MessagePattern, Payload, Transport, MicroserviceOptions } from "@nestjs/microservices";

@Controller()
class ProductsController {
  @MessagePattern("products.get")
  get(@Payload() data: { id: number }) {
    return { id: data.id, name: `Product #${data.id}`, price: 42, servedBy: `products-${process.pid}` };
  }
}

@Module({ controllers: [ProductsController] })
class ProductsModule {}

async function bootstrap() {
  const app = await NestFactory.createMicroservice<MicroserviceOptions>(ProductsModule, {
    transport: Transport.NATS,
    options: { servers: ["nats://localhost:4222"], queue: "products" }, // queue group = load balancing
  });
  await app.listen();
  console.log(`products microservice up (pid ${process.pid})`);
}
bootstrap();

The gateway is an ordinary Nest HTTP app with a ClientProxy injected. Note the two things you have to do yourself: convert the URL parameter with a pipe, and put a timeout on the call with RxJS.

// nest/gateway.ts — a NestJS HTTP app that calls the products microservice over NATS.
import "reflect-metadata";
import { Controller, Get, Module, Param, ParseIntPipe } from "@nestjs/common";
import { NestFactory } from "@nestjs/core";
import { ClientsModule, ClientProxy, Transport } from "@nestjs/microservices";
import { Inject } from "@nestjs/common";
import { firstValueFrom, timeout } from "rxjs";

@Controller("products")
class ProductsGatewayController {
  constructor(@Inject("PRODUCTS") private readonly client: ClientProxy) {}

  @Get(":id")
  get(@Param("id", ParseIntPipe) id: number) {
    // send() returns an Observable; you add timeouts/retries yourself.
    return firstValueFrom(this.client.send("products.get", { id }).pipe(timeout(2000)));
  }
}

@Module({
  imports: [
    ClientsModule.register([
      { name: "PRODUCTS", transport: Transport.NATS, options: { servers: ["nats://localhost:4222"] } },
    ]),
  ],
  controllers: [ProductsGatewayController],
})
class GatewayModule {}

async function bootstrap() {
  const app = await NestFactory.create(GatewayModule, { logger: ["error", "warn"] });
  await app.listen(process.env.PORT || 3000);
  console.log(`gateway on http://localhost:${process.env.PORT || 3000}`);
}
bootstrap();

Run two service instances and the gateway, then hit it:

$ npx tsx nest/products.service.ts &
$ npx tsx nest/products.service.ts &
$ npx tsx nest/gateway.ts &
$ for i in 1 2 3 4; do curl -s localhost:3000/products/$i; echo; done
{"id":1,"name":"Product #1","price":42,"servedBy":"products-3460437"}
{"id":2,"name":"Product #2","price":42,"servedBy":"products-3460417"}
{"id":3,"name":"Product #3","price":42,"servedBy":"products-3460437"}
{"id":4,"name":"Product #4","price":42,"servedBy":"products-3460437"}

$ curl -s localhost:3000/products/abc
{"message":"Validation failed (numeric string is expected)","error":"Bad Request","statusCode":400}

Load balancing works (NATS is doing it, not Nest — the queue group hands each message to one subscriber). Validation works because we added ParseIntPipe. Now stop both products processes and call again:

$ curl -s localhost:3000/products/5
{"statusCode":500,"message":"Internal server error"}
# gateway log: EmptyResponseException: Empty response. There are no subscribers listening to that message ("products.get")

Fast failure, which is good — NATS reports "no responders". But it is a generic 500, there is no retry, no circuit breaker, and nothing in the gateway knows that products exists at all except the string "products.get".

Moleculer

The same service as a Moleculer service schema. Validation is declared on the action; convert: true handles the string-from-URL case.

// moleculer/products.service.js — the same service in Moleculer.
const { ServiceBroker } = require("moleculer");

const broker = new ServiceBroker({
  nodeID: `products-${process.pid}`,
  transporter: "nats://localhost:4222",
  logger: false,
});

broker.createService({
  name: "products",
  actions: {
    get: {
      params: { id: { type: "number", convert: true } },   // validation is part of the action
      handler(ctx) {
        return { id: ctx.params.id, name: `Product #${ctx.params.id}`, price: 42, servedBy: this.broker.nodeID };
      },
    },
  },
});

broker.start().then(() => console.log(`products service up (${broker.nodeID})`));

The gateway uses moleculer-web, which is a service that maps HTTP routes to actions. The timeout is a broker option, so it applies to every call in the process.

// moleculer/gateway.js — HTTP edge, mapping a route to the action.
const { ServiceBroker } = require("moleculer");
const ApiGateway = require("moleculer-web");

const broker = new ServiceBroker({
  nodeID: `gateway-${process.pid}`,
  transporter: "nats://localhost:4222",
  logger: false,
  requestTimeout: 2000,                       // global; NestJS needs rxjs timeout() per call
});

broker.createService({
  name: "api",
  mixins: [ApiGateway],
  settings: {
    port: process.env.PORT || 3000,
    routes: [{ path: "/", aliases: { "GET /products/:id": "products.get" } }],
  },
});

broker.start().then(() => console.log(`gateway on http://localhost:${process.env.PORT || 3000}`));

$ node moleculer/products.service.js &
$ node moleculer/products.service.js &
$ node moleculer/gateway.js &
$ for i in 1 2 3 4; do curl -s localhost:3000/products/$i; echo; done
{"id":1,"name":"Product #1","price":42,"servedBy":"products-3460597"}
{"id":2,"name":"Product #2","price":42,"servedBy":"products-3460596"}
{"id":3,"name":"Product #3","price":42,"servedBy":"products-3460597"}
{"id":4,"name":"Product #4","price":42,"servedBy":"products-3460596"}

$ curl -s localhost:3000/products/abc
{"name":"ValidationError","message":"Parameters validation error!","code":422,"type":"VALIDATION_ERROR","data":[{"type":"number","message":"The 'id' field must be a number.","field":"id","actual":"abc","nodeID":"gateway-3460598","action":"products.get"}]}

# stop both products processes, then:
$ curl -s localhost:3000/products/5
{"name":"ServiceNotAvailableError","message":"Service 'products.get' is not available.","code":404,"type":"SERVICE_NOT_AVAILABLE","data":{"action":"products.get"}}

Here the load balancing is done by Moleculer's registry (round-robin by default; you can switch to latency-based, CPU-based or sharded). When the service instances went away, the gateway knew — it has a live registry of every node and action in the cluster, updated by heartbeats — and returned a structured, typed error. Add retryPolicy and circuitBreaker to the broker options and you get retries and a breaker on every call with no code changes; the previous article shows both in action.

What you would have to add to NestJS

If you build a multi-service system on @nestjs/microservices, here is what is not in the box and what people typically bolt on:

Service registry / discovery — nothing. You rely on the broker's semantics (NATS subjects, Kafka topics) or on Kubernetes DNS. There is no way to ask "which services are online, on which nodes?"
Load balancing — only what the transport gives you (NATS queue groups, Kafka consumer groups). TCP transport: one address, no balancing.
Retries, circuit breaker, bulkhead, fallback — nothing built in. opossum, RxJS retry(), or a service mesh.
Timeouts — per call, with RxJS timeout().
Payload validation on the message side — ValidationPipe with class-validator DTOs, same as HTTP.
Caching of RPC results — @nestjs/cache-manager on the HTTP side; nothing at the transport level.
Metrics & tracing across service calls — community OpenTelemetry packages; you propagate context yourself.
Local vs remote transparency — a ClientProxy always goes through the broker, even to a handler in the same process. There is no "start as a monolith, split later" mode.

None of this is a criticism. NestJS's microservices package is a transport abstraction — it makes a Nest app able to talk over a broker with the same controller idioms as HTTP. That is exactly what it claims to be. It is just not a service layer, and the difference matters a lot once you have ten services and a bad Tuesday.

What you would have to add to Moleculer

The other direction is just as real:

TypeScript ergonomics. Moleculer ships type definitions and works fine in TypeScript, but its native idiom is the schema object, not decorators and classes. Community packages (moleculer-decorators, TypeScript project templates) close some of the gap; the DX is not at NestJS's level, and I'd rather say that than have you find out later. The 0.15 release invested in typings (typed service lifecycle handlers, middleware, error handlers), and it keeps improving.
Dependency injection. Moleculer has mixins and service-level this for shared logic, not a DI container. If your team's mental model is Angular-style providers, you will miss it.
Application structure. Nest's CLI scaffolds modules, tests and DTOs and enforces a layout. Moleculer has a project generator, but inside a service you organise the code however you like.
Ecosystem breadth. NestJS has an official module for nearly every integration. Moleculer has an API gateway, database adapters, channels (queue-based reliable events), Socket.IO, GraphQL — and then you're writing your own service around an npm client, which is easy but is yours to maintain.
Community size and hiring. This is the honest big one. NestJS has ~76k GitHub stars and ~54 million monthly npm downloads; Moleculer has ~6k stars and ~250k downloads. A NestJS job ad gets more applicants who have used it. If you are building a team of twenty, that difference is worth real money.

Can you use both?

Yes, and it is more common than you'd think. The combinations that work:

Moleculer as the internal service layer, NestJS at the edge. Nest handles HTTP with its full toolkit (guards, Swagger, GraphQL), and a Moleculer broker running inside the Nest process calls the services. You get Nest's DX where users touch the system and Moleculer's registry, balancing and resilience behind it.
NestJS everywhere, Moleculer for one hairy subsystem that needs pub/sub with balanced consumers, sharding or circuit breaking that you don't want to hand-roll.

What doesn't work well is running both frameworks' microservices transports side by side for the same calls — pick one owner for inter-service communication.

Which one, and when

Pick NestJS if:

You are building one service, or a monolith that might split someday. Most apps are this. Nest is the better single-app framework, full stop.
Your team is TypeScript-native and values decorators, DI and enforced structure.
You need the breadth: GraphQL federation, WebSocket gateways, Swagger generation, Passport strategies, all official.
Hiring and onboarding speed matter more than the distributed-systems layer.
You already run Kubernetes with a service mesh, so discovery and resilience are the platform's job.

Pick Moleculer if:

You are building a system of many services now, and you do not want to assemble discovery, balancing, retries, circuit breakers, caching and tracing from six separate packages.
You do not have Kubernetes or a service mesh and don't want to run one — Moleculer gives you those properties on a VM with Docker Compose (see Node.js microservices without Kubernetes).
Your architecture is event-driven — balanced and broadcast events across services, over NATS, Redis, Kafka or MQTT, are first-class.
You want to start as a modular monolith and split into processes later without changing service code.
Plain JavaScript is fine, or you're happy with Moleculer's level of TypeScript support.

And if you're undecided: build the first service in NestJS. If the day comes when you have five of them calling each other and you find yourself writing a retry helper for the third time, that's the moment to look at Moleculer — and the boundary between the two is clean enough that you can.

Code in this article was run against NestJS 12.0 and Moleculer 0.15.2 on Node.js 22 with NATS 2. The full example project, including scripts that reproduce the outputs above, is in the moleculer-examples repository.

Node.js microservices without Kubernetes (and without a platform team)

Icebob — Mon, 07 Sep 2026 18:50:32 +0000

Somewhere around 2018 the phrase "we're doing microservices" silently became "we're running Kubernetes". The two ideas fused. Today when a Node.js team asks how to split a backend into services, the answer they get is a shopping list: a cluster, an ingress controller, a service mesh for retries and mTLS, Consul or CoreDNS for discovery, Prometheus and Jaeger wired into sidecars, and a Helm chart per service to hold it all together.

That stack is excellent. It is also, for most teams that are not Google, the wrong first step. This article is about the other path: getting the actual properties of microservices — independent deployment, service discovery, load balancing, fault isolation, an event bus — in plain Node.js processes, on a single VM or a handful of them, without an orchestrator.

I maintain Moleculer, a microservices framework for Node.js and TypeScript that was built for exactly this, so the examples use it. But the argument comes first, and it holds regardless of which tool you pick.

What you actually need from "microservices"

Strip away the branding and a microservice architecture is a short list of capabilities:

Service discovery — service A needs to find a live instance of service B without hardcoding IPs.
Load balancing — if B has three instances, calls should spread across them, and a dead instance should stop receiving traffic.
Inter-service communication — request/response (RPC) and publish/subscribe (events), with serialization handled for you.
Resilience — timeouts, retries, circuit breakers, bulkheads and fallbacks so one slow dependency doesn't take the whole system down.
Observability — metrics and distributed tracing across service boundaries.
Independent deployment — ship service B without redeploying A.

Kubernetes plus a service mesh gives you all of that at the infrastructure level: discovery via DNS and Services, balancing via kube-proxy or Envoy, resilience via Istio/Linkerd policies, tracing via sidecars. It is a legitimate design. It's just a lot of moving parts, and every one of them is something your team has to run, upgrade and debug at 3am.

The alternative is to put those capabilities in the application layer — in a library your services import — and let the infrastructure be as dumb as a VM with Docker Compose. That's the Moleculer model.

Moleculer in sixty seconds

A few terms, each glossed with what you'd type into a search box:

ServiceBroker — the runtime and service registry that lives in every process. It knows which services exist, where they are, and routes calls to them.
Service — a named object with actions (RPC endpoints, callable as "orders.create") and event handlers (subscribers).
Transporter — the message broker / transport layer between processes: NATS, Redis, Kafka, MQTT, AMQP or plain TCP. Swappable with one config line.

The crucial property: a service's code does not know whether the service it's calling is in the same process or on another machine. ctx.call("products.get") is the same line either way.

Step 1: three services, one process

Let's write a tiny shop: products, orders and notifications. Here is the business logic, and note that this file never changes for the rest of the article.

// services.js — the business logic. It never changes between "one process" and "many processes".

const Products = {
  name: "products",
  actions: {
    get: {
      params: { id: { type: "number", convert: true } }, // convert: URL params arrive as strings
      handler(ctx) {
        return { id: ctx.params.id, name: `Product #${ctx.params.id}`, price: 42, servedBy: this.broker.nodeID };
      },
    },
  },
};

const Orders = {
  name: "orders",
  actions: {
    create: {
      params: { productId: "number", qty: "number" },
      async handler(ctx) {
        // Looks like a local function call — may be a network hop. The framework decides.
        const product = await ctx.call("products.get", { id: ctx.params.productId });
        const order = { id: Date.now(), product, qty: ctx.params.qty, total: product.price * ctx.params.qty };
        // Publish an event. Whoever cares subscribes.
        await ctx.emit("order.created", order);
        return order;
      },
    },
  },
};

const Notifications = {
  name: "notifications",
  events: {
    "order.created"(ctx) {
      this.logger.info(`Email sent: order ${ctx.params.id}, total ${ctx.params.total} (from ${ctx.nodeID})`);
    },
  },
};

module.exports = { Products, Orders, Notifications };

The params block is a validation schema (Moleculer ships with fastest-validator); an invalid call is rejected with a structured ValidationError before your handler runs.

Now load all three into a single broker and run it. No message broker, no Docker, nothing listening on a port:

// monolith.js — all three services in ONE process. No broker, no network, no Docker.
const { ServiceBroker } = require("moleculer");
const { Products, Orders, Notifications } = require("./services");

const broker = new ServiceBroker({ logger: { type: "Console", options: { level: "info" } } });

broker.createService(Products);
broker.createService(Orders);
broker.createService(Notifications);

broker.start()
  .then(() => broker.call("orders.create", { productId: 7, qty: 3 }))
  .then((order) => console.log("Order:", order))
  .then(() => broker.stop());

$ npm install moleculer
$ node monolith.js
INFO  dev-laptop-51203/NOTIFICATIONS: Email sent: order 1788453205755, total 126 (from dev-laptop-51203)
Order: {
  id: 1788453205755,
  product: { id: 7, name: 'Product #7', price: 42, servedBy: 'dev-laptop-51203' },
  qty: 3,
  total: 126
}

This is a modular monolith: one deployable, but the modules only talk through the same contract they'd use over a network. It is a perfectly good place to stay for a long time. Many Moleculer users never leave it.

Step 2: split into processes with one config line

To run the services in separate processes, add a transporter. We'll use NATS — a single ~20 MB binary with no dependencies, and the best default for most teams.

$ docker run -d -p 4222:4222 nats:2
$ npm install nats

The node script takes the names of the services it should host from the command line:

// node.js — start ONE process that hosts the services named on the command line.
//   node node.js products
//   node node.js orders notifications
// Same service code as before. The only new thing is the transporter.
const { ServiceBroker } = require("moleculer");
const all = require("./services");

const broker = new ServiceBroker({
  nodeID: `${process.argv.slice(2).join("+")}-${process.pid}`,
  transporter: process.env.TRANSPORTER || "nats://localhost:4222",
  logger: { type: "Console", options: { level: "info" } },
});

for (const name of process.argv.slice(2)) {
  const schema = Object.values(all).find((s) => s.name === name);
  if (!schema) throw new Error(`Unknown service: ${name}`);
  broker.createService(schema);
}

broker.start();

Start two of them in two terminals:

$ node node.js products
$ node node.js orders notifications

And a client that has no services of its own — it just joins the cluster and calls in:

// client.js — a broker with no services of its own; it just calls into the cluster.
const { ServiceBroker } = require("moleculer");

const broker = new ServiceBroker({
  nodeID: "client",
  transporter: process.env.TRANSPORTER || "nats://localhost:4222",
  logger: false,
});

broker.start()
  .then(() => broker.waitForServices(["orders"]))
  .then(async () => {
    for (let i = 1; i <= 4; i++) {
      const order = await broker.call("orders.create", { productId: i, qty: 2 });
      console.log(`order ${order.id} → total ${order.total}, product served by ${order.product.servedBy}`);
    }
    // Built-in introspection: which nodes are online and what do they host?
    const nodes = await broker.call("$node.list", { withServices: true });
    for (const n of nodes) console.log(`node ${n.id}: ${n.services.map((s) => s.name).filter((s) => s !== "$node").join(", ")}`);
  })
  .then(() => broker.stop());

$ node client.js
order 1788453251083 → total 84, product served by products-3455407
order 1788453251095 → total 84, product served by products-3455407
order 1788453251104 → total 84, product served by products-3455407
order 1788453251112 → total 84, product served by products-3455407
node client:
node orders+notifications-3455409: orders, notifications
node products-3455407: products

Look at what just happened without any configuration:

The orders process called products.get in a different process. Nobody told it where products lives.
The order.created event crossed the network and reached notifications (check the second terminal — it logged the four emails).
The client discovered the whole cluster via the built-in $node service — a service registry you get for free.

That is service discovery. There is no Consul, no etcd, no DNS trick. When a node starts it announces itself over the transporter; every other broker updates its local registry; heartbeats detect when a node disappears. The registry lives inside each process, so a lookup is a hash-map read, not a network call.

Step 3: scale a service by starting another copy

Open a third terminal and start a second products process. Then run the client again:

$ node node.js products      # second instance
$ node client.js
order 1788453267932 → total 84, product served by products-3455456
order 1788453267948 → total 84, product served by products-3455458
order 1788453267956 → total 84, product served by products-3455456
order 1788453267965 → total 84, product served by products-3455458
node orders+notifications-3455460: orders, notifications
node products-3455456: products
node products-3455458: products

Calls now alternate between the two instances. That's the default round-robin load balancing; you can switch to random, CPU-usage-based, latency-based or shard-by-key strategies with one option. Stop one of the products processes and traffic moves to the survivor: a graceful stop deregisters the node immediately, and a hard crash is caught by the heartbeat timeout (configurable; the balancing docs cover the strategies and the fault-tolerance docs the failover behaviour).

Notice the shape of the "deploy": starting a process. Scaling: starting another process. There is no Deployment manifest, no replica count to reconcile, no rollout to watch. If your team runs services with systemd, PM2 or Docker Compose, you already have everything the framework needs.

Resilience is a broker option, not a library

In the Kubernetes world, retries, timeouts and circuit breaking usually arrive with the service mesh. Without a mesh you'd reach for opossum, p-retry and AbortSignal.timeout and glue them into every call site. Moleculer bakes them into the call path:

// resilience.js — timeout, retry and circuit breaker are broker options, not extra libraries.
const { ServiceBroker, Errors } = require("moleculer");

const broker = new ServiceBroker({
  logger: false,
  requestTimeout: 500,                    // ms; a hanging call becomes a RequestTimeoutError
  retryPolicy: { enabled: true, retries: 2, delay: 50, factor: 2 },
  circuitBreaker: {
    enabled: true,
    threshold: 0.5,                       // open when >50% of calls fail...
    minRequestCount: 5,                   // ...after at least 5 calls
    windowTime: 60,                       // seconds
    halfOpenTime: 5_000,                  // ms until we probe again
  },
});

let calls = 0;
broker.createService({
  name: "payments",
  actions: {
    charge: {
      retryPolicy: { enabled: false },     // retries make no sense for a non-idempotent charge
      handler() {
        calls++;
        throw new Errors.MoleculerError("gateway down", 503); // 5xx errors count against the breaker
      },
    },
    fallbackDemo: {
      fallback: () => ({ cached: true }),  // return this instead of throwing
      handler() { throw new Errors.MoleculerError("nope", 500); },
    },
  },
});

broker.start().then(async () => {
  for (let i = 1; i <= 8; i++) {
    try { await broker.call("payments.charge", { amount: 10 }); }
    catch (err) { console.log(`call ${i}: ${err.name} (${err.message})`); }
  }
  console.log("handler actually ran:", calls, "times");
  console.log("fallback:", await broker.call("payments.fallbackDemo"));
  await broker.stop();
});

$ node resilience.js
call 1: MoleculerError (gateway down)
call 2: MoleculerError (gateway down)
call 3: MoleculerError (gateway down)
call 4: MoleculerError (gateway down)
call 5: MoleculerError (gateway down)
call 6: ServiceNotAvailableError (Service 'payments.charge' is not available.)
call 7: ServiceNotAvailableError (Service 'payments.charge' is not available.)
call 8: ServiceNotAvailableError (Service 'payments.charge' is not available.)
handler actually ran: 5 times
fallback: { cached: true }

After five failures the circuit breaker opened and the next calls failed fast without touching the handler — exactly what you want when a payment gateway is melting down. Retries and timeouts are configured globally and overridden per action (note retryPolicy: { enabled: false } on the non-idempotent charge). A bulkhead (per-action concurrency limit) is one more option. None of this needed a sidecar.

Putting an HTTP door on it

Services talk to each other over the transporter, but browsers speak HTTP. The moleculer-web package is an API gateway that is itself a service — it joins the cluster like any other node and maps routes to actions:

// gateway.js — HTTP edge. It is just another node in the cluster; it owns no business logic.
const { ServiceBroker } = require("moleculer");
const ApiGateway = require("moleculer-web");

const broker = new ServiceBroker({
  nodeID: `gateway-${process.pid}`,
  transporter: process.env.TRANSPORTER || "nats://localhost:4222",
  logger: { type: "Console", options: { level: "warn" } },
});

broker.createService({
  name: "api",
  mixins: [ApiGateway],
  settings: {
    port: process.env.PORT || 3000,
    routes: [{
      path: "/api",
      // Only expose what you list here; everything else stays internal to the cluster.
      whitelist: ["products.get", "orders.create"],
      aliases: {
        "GET /products/:id": "products.get",
        "POST /orders": "orders.create",
      },
      bodyParsers: { json: true },
    }],
  },
});

broker.start().then(() => console.log(`Gateway listening on http://localhost:${process.env.PORT || 3000}/api`));

$ npm install moleculer-web
$ node gateway.js
$ curl localhost:3000/api/products/7
{"id":7,"name":"Product #7","price":42,"servedBy":"products-3455667"}

$ curl -X POST localhost:3000/api/orders -H 'content-type: application/json' -d '{"productId":7,"qty":2}'
{"id":1788453318747,"product":{"id":7,"name":"Product #7","price":42,"servedBy":"products-3455612"},"qty":2,"total":84}

$ curl -X POST localhost:3000/api/orders -H 'content-type: application/json' -d '{"productId":"x"}'
{"name":"ValidationError","message":"Parameters validation error!","code":422,"type":"VALIDATION_ERROR","data":[{"type":"number","message":"The 'productId' field must be a number.","field":"productId","actual":"x","nodeID":"gateway-3455616","action":"orders.create"},{"type":"required","message":"The 'qty' field is required.","field":"qty","nodeID":"gateway-3455616","action":"orders.create"}]}

The gateway has no knowledge of where products runs. The validation error came from the schema on the service, propagated back through the transporter, and was turned into a 422 automatically. Run two gateways behind nginx or Caddy and you've got a redundant edge.

Deploying without an orchestrator

Here is the entire production topology for a small team, as a Compose file. One VM (or one per service if you like), a NATS container, and your Node processes:

# docker-compose.yml — the whole "cluster" on one VM. No orchestrator.
services:
  nats:
    image: nats:2

  products:
    build: .
    command: node node.js products
    environment: { TRANSPORTER: nats://nats:4222 }
    depends_on: [nats]
    deploy: { replicas: 2 }          # two instances; the framework balances between them

  orders:
    build: .
    command: node node.js orders notifications
    environment: { TRANSPORTER: nats://nats:4222 }
    depends_on: [nats]

  gateway:
    build: .
    command: node gateway.js
    environment: { TRANSPORTER: nats://nats:4222 }
    ports: ["3000:3000"]
    depends_on: [nats]

docker compose up -d and you have a load-balanced, self-discovering, fault-tolerant service cluster. To spread it over several machines, point every TRANSPORTER at the same NATS host — NATS itself clusters trivially if you need it to be redundant. Metrics (Prometheus, StatsD, Datadog) and distributed tracing (Jaeger, Zipkin, OpenTelemetry) are broker options too, so observability doesn't need sidecars either.

What you give up compared to Kubernetes, honestly:

Automatic rescheduling. If a VM dies, nothing restarts your containers elsewhere. You handle it with a second VM and a health check, or accept the risk.
Rolling deploys as a primitive. Compose replaces containers; for zero-downtime you rely on Moleculer's graceful shutdown (a stopping node deregisters first and, with request tracking enabled, waits for in-flight requests before exiting) plus at least two replicas.
Autoscaling. You scale by editing replicas. For most teams that's a feature.
Secrets, RBAC, network policy. You use your OS and cloud provider's tools instead.

When you do want Kubernetes

If you already run a cluster, keep it — Moleculer runs perfectly well inside pods. The two layers are complementary rather than competing: Kubernetes handles scheduling, restarts and rollouts; Moleculer handles discovery, balancing and resilience at the application level, which means you don't need a service mesh on top. The official Kubernetes guide covers it.

You should also reach for an orchestrator when you have a genuinely large fleet (dozens of services, hundreds of instances), strict multi-tenant isolation requirements, or a platform team whose job it is to run one.

But if you are three developers with a product to ship, the honest answer to "how do we do microservices in Node.js?" is: one message broker, a framework that owns the distributed-systems layer, and processes you start with a command you already know. Start as a modular monolith, split when a boundary is real, scale by launching another copy. Get to Kubernetes when Kubernetes solves a problem you actually have.

All the code above runs as shown against Moleculer 0.15 on Node.js 22. The full example project is at github.com/moleculerjs/moleculer-examples — docs at moleculer.services.

DEV Community: Icebob

Moleculer vs gRPC vs tRPC: an RPC protocol is not a service layer

What each one is

The same service, three times

Problem 1: a second instance

Problem 2: an instance dies

Problem 3: an instance is slow

Problem 4: a third instance appears

Type safety, honestly

Other languages, browsers, streaming

Throughput shape

Using them together

tRPC in front of a Moleculer cluster

A gRPC service behind a Moleculer action

Which one, when

Event-driven architecture in Node.js: pub/sub with NATS, Kafka and Redis — and when an event is not enough

The setup: one producer, two kinds of consumers

1. emit is balanced, broadcast is not

2. Groups and wildcards

3. Validation happens at the subscriber

4. Same code, three brokers

5. What happens when the subscriber is down

Which one, when

Circuit breaker, retry and timeout in Node.js microservices — and why you need all three

The setup: a dependency with knobs

1. Retry: try again, but only when it makes sense

2. Circuit breaker: stop asking a node that keeps saying no

3. Timeout: the one that is off by default

4. Distributed timeout: the budget travels with the request

5. Fallback: degrade instead of failing

6. Bulkhead: don't let one endpoint sink the ship

What to put in production

Service discovery and load balancing in Node.js — without Consul or Kubernetes

How it works, in one paragraph

The setup

1. Three workers, zero configuration

2. Scaling down, crashing, scaling up — with traffic running

3. Choosing who gets the call: strategies

Writing your own in fifteen lines

One more rule you get for free: preferLocal

4. No message broker either: discovery over plain TCP

When you still want Consul or Kubernetes

From a modular monolith to microservices without a rewrite

The starting point

Stage 1: module boundaries become service boundaries — same process

Stage 2: extract one service — the one that actually needs it

Stage 3: split the rest, scale one — and meet the thing that breaks

The migration order that works

Using Moleculer with TypeScript: contracts, two service styles, and the honest caveats

What Moleculer gives you out of the box

Step 1: a contract file

Step 2: a service in schema style

Step 3: a service in class style

Step 4: a typed caller

Step 5: running it

The honest caveats

Recommendations

Moleculer vs NestJS: which one, and when

First, what each one is

The same service, twice

NestJS

Moleculer

What you would have to add to NestJS

What you would have to add to Moleculer

Can you use both?

Which one, and when

Node.js microservices without Kubernetes (and without a platform team)

What you actually need from "microservices"

Moleculer in sixty seconds

Step 1: three services, one process

Step 2: split into processes with one config line

Step 3: scale a service by starting another copy

Resilience is a broker option, not a library

Putting an HTTP door on it

Deploying without an orchestrator

When you do want Kubernetes

1. `emit` is balanced, `broadcast` is not

One more rule you get for free: `preferLocal`