DEV Community

Cover image for Beyond Kafka and Redis: A Practical Guide to NATS as Your Unified Cloud-Native Backbone
Dave Maman
Dave Maman

Posted on Edited on

Beyond Kafka and Redis: A Practical Guide to NATS as Your Unified Cloud-Native Backbone

#natsio #redis #kafka #pubsub

Updated October 2026: fixed three code errors (KV TTL units, KV key names, and the optimistic-locking option) and tightened the wording. Part 2 builds an AI chat backend with the features NATS added in 2.11 through 2.15.

A common platform setup looks like this. Kafka handles event streaming, with its KRaft controllers and partition planning. Redis Sentinel handles sessions and rate limits. S3 or MinIO stores model weights and config files. Each system has its own runbook, its own dashboards, and its own ways to fail.

When you get paged at 3 AM, you're debugging that infrastructure as well as your application.

NATS can cover most of that stack with one server binary. It's been running large distributed systems for over a decade. With JetStream, the Key-Value store, and the Object Store now mature, it's a full messaging platform as well as a fast pub/sub system.

This post walks through each piece with working code.

What NATS replaces

Need Usual tool NATS feature
Pub/sub messaging Redis Pub/Sub, Kafka topics Core NATS subjects
Persistent streaming Kafka, Pulsar, Kinesis JetStream streams
Key-value storage Redis, etcd, Consul KV KV store
Object and blob storage S3, MinIO Object Store
Request-reply between services HTTP, gRPC Built-in request-reply
Service discovery Consul, Eureka, Kubernetes DNS Services framework

The operational difference is large. A production Kafka deployment usually means:

  • A KRaft controller quorum (Kafka 4.0 removed ZooKeeper; older clusters still run it)
  • Broker nodes with partition and replication planning
  • A schema registry for Avro or Protobuf
  • Kafka Connect for data pipelines
  • JMX exporters for monitoring

A NATS server with persistence is one command:

nats-server -js
Enter fullscreen mode Exit fullscreen mode

There are no external dependencies. The -js flag turns on JetStream.

Core NATS

Core NATS moves messages between publishers and subscribers on named subjects, like events.user.signup. Nothing is stored at this layer. If no subscriber is listening when a message arrives, the message is dropped. In exchange, latency is typically under a millisecond.

Start the server 

# Run the binary
nats-server

# Or with Docker
docker run -p 4222:4222 nats:latest

# Turn on JetStream for persistence
nats-server -js
Enter fullscreen mode Exit fullscreen mode

Publish and subscribe 

import { connect } from "@nats-io/transport-node";

const nc = await connect({ servers: "localhost:4222" });

// Subscriber (usually another process)
const sub = nc.subscribe("events.user.*");
(async () => {
  for await (const msg of sub) {
    console.log(`Received on ${msg.subject}:`, msg.string());
  }
})();

// Publisher
nc.publish("events.user.signup", JSON.stringify({
  userId: "user-123",
  email: "dev@example.com",
  timestamp: Date.now(),
}));
Enter fullscreen mode Exit fullscreen mode

events.user.* matches one token in that position, so the subscriber gets events.user.signup, events.user.login, and so on. > matches one or more tokens at the end of a subject.

With Kafka, the same flow needs a topic created with partitions and a replication factor, producer serializers, and a consumer group that handles partition assignment and rebalancing. With NATS you publish to a subject and subscribe to it.

Request-reply

NATS has request-reply built in, so internal services can call each other without HTTP or gRPC:

// Service: answer requests
const svc = nc.subscribe("api.users.get");
(async () => {
  for await (const msg of svc) {
    const { userId } = msg.json<{ userId: string }>();
    const user = await db.findUser(userId);
    msg.respond(JSON.stringify(user));
  }
})();

// Client: send a request and wait for the reply
const response = await nc.request(
  "api.users.get",
  JSON.stringify({ userId: "user-123" }),
  { timeout: 5000 },
);
console.log("User:", response.json());
Enter fullscreen mode Exit fullscreen mode

If several instances subscribe with the same queue group (nc.subscribe("api.users.get", { queue: "users" })), NATS sends each request to one of them. That's load balancing without a load balancer.

For AI workloads: request-reply fits synchronous inference. The application sends a prompt to inference.chat.query, one model server in the queue group takes it, and the reply comes back on the same connection. Add model servers and they share the load.

JetStream: persistent streams

JetStream is the persistence layer. A stream stores every message published to its subjects, and consumers read from it, with acknowledgment and replay. It runs inside the same server, so there's no second cluster to manage.

Create a stream

With the CLI:

nats stream add ORDERS \
  --subjects "orders.*" \
  --retention limits \
  --max-msgs 1000000 \
  --max-bytes 1GB \
  --max-age 7d \
  --defaults
Enter fullscreen mode Exit fullscreen mode

--defaults accepts the default for every setting you didn't pass. Without it, the CLI prompts for each one.

Or in code:

import { connect } from "@nats-io/transport-node";
import { AckPolicy, DeliverPolicy, jetstream, jetstreamManager } from "@nats-io/jetstream";

const nc = await connect({ servers: "localhost:4222" });
const jsm = await jetstreamManager(nc);

await jsm.streams.add({
  name: "ORDERS",
  subjects: ["orders.*"],
  retention: "limits",
  max_msgs: 1_000_000,
  max_bytes: 1024 * 1024 * 1024, // 1 GB
  max_age: 7 * 24 * 60 * 60 * 1_000_000_000, // 7 days, in nanoseconds
});
Enter fullscreen mode Exit fullscreen mode

Stream durations like max_age are in nanoseconds. Remember that when you get to the KV section, because nats.js KV options use milliseconds.

Publish to a stream 

const js = jetstream(nc);

const ack = await js.publish("orders.created", JSON.stringify({
  orderId: "ORD-12345",
  customerId: "CUST-789",
  items: [{ sku: "WIDGET-01", quantity: 2 }],
  total: 49.99,
}));
console.log(`Stored in ${ack.stream} at sequence ${ack.seq}`);

// Deduplicate retries with a message ID
await js.publish("orders.created", JSON.stringify({ orderId: "ORD-12346" }), {
  msgID: "ORD-12346",
});
Enter fullscreen mode Exit fullscreen mode

A JetStream publish waits for the server to confirm it stored the message. With msgID, a retry of the same publish inside the stream's duplicate window (two minutes by default) is dropped instead of stored twice. Delivery to consumers is at-least-once. For exactly-once processing, combine msgID on publish with ackAck() on the consumer, which waits for the server to confirm the ack.

Consume with a durable consumer 

await jsm.consumers.add("ORDERS", {
  durable_name: "order-processor",
  ack_policy: AckPolicy.Explicit,
  deliver_policy: DeliverPolicy.All, // start from the first message
  filter_subject: "orders.created",
  max_ack_pending: 100,
});

const consumer = await js.consumers.get("ORDERS", "order-processor");
const messages = await consumer.consume();

for await (const msg of messages) {
  const order = msg.json<{ orderId: string }>();
  console.log(`Processing order ${order.orderId}`);
  await processOrder(order);
  msg.ack();
}
Enter fullscreen mode Exit fullscreen mode

A durable consumer keeps its position on the server. If the process restarts, it continues from the last acknowledged message. A message that isn't acknowledged in time is delivered again.

Check a stream from the CLI:

nats stream info ORDERS
Enter fullscreen mode Exit fullscreen mode 
State:
              Messages: 15,234
                 Bytes: 2.1 MiB
        First Sequence: 1 @ 2026-01-15 10:30:00
         Last Sequence: 15,234 @ 2026-01-15 14:22:31
Enter fullscreen mode Exit fullscreen mode

For AI workloads: JetStream works well as a durable log of agent actions. Publish each decision to agent.<agent-id>.actions, and you can replay the exact sequence later when you debug why an agent did something. You can also queue inference requests in a stream, so a model server restart doesn't lose them.

Key-Value store

The KV store is built on JetStream. You get a key-value API, plus per-key history, watchers, and TTL.

import { Kvm } from "@nats-io/kv";

const kv = await new Kvm(nc).create("sessions", {
  history: 5, // keep the last 5 values of each key
  ttl: 60 * 60 * 1000, // 1 hour, in milliseconds
});

await kv.put("user-123", JSON.stringify({
  userId: "user-123",
  role: "admin",
  lastActivity: Date.now(),
}));

const entry = await kv.get("user-123");
console.log("Session:", entry?.string());
console.log("Revision:", entry?.revision); // increases on every write

// Optimistic locking: write only if nobody changed the key since we read it
try {
  await kv.update("user-123", JSON.stringify({ role: "viewer" }), entry!.revision);
} catch {
  console.log("Someone else changed this session first");
}

// Watch for changes
const watch = await kv.watch();
for await (const e of watch) {
  console.log(`${e.key}: ${e.operation}`);
}
Enter fullscreen mode Exit fullscreen mode

Keys can use letters, digits, -, _, /, =, and .. A colon isn't allowed, so the Redis habit of session:user-123 throws an invalid key error. Use dots instead (session.user-123). Dots also let you watch a group of keys with a wildcard, like session.>.

Compared with Redis:

  • Redis needs Sentinel or Cluster for high availability. A KV bucket replicates like any stream (replicas: 3).
  • Redis keeps one value per key. KV keeps as many past values as you set in history.
  • Watching keys in Redis means keyspace notifications or Pub/Sub. KV has watchers built in.
  • Both have TTL, but only KV gives every write a revision you can use for optimistic locking.

From the CLI:

nats kv put sessions user-456 '{"userId":"user-456"}'
nats kv get sessions user-456
nats kv history sessions user-456
Enter fullscreen mode Exit fullscreen mode

For AI workloads: KV fits conversation state. Store the context window, tool results, and memory under a key like agent.<session-id>.state. History shows how the state changed, watchers let other components react to changes, and the bucket TTL removes abandoned sessions.

Object Store

The Object Store keeps files too large for a single message. It splits each file into chunks stored in a stream, and records a SHA-256 digest so readers can check integrity.

import { readFile } from "node:fs/promises";
import { Objm, StorageType } from "@nats-io/obj";

const store = await new Objm(nc).create("models", {
  storage: StorageType.File,
  description: "Model artifacts",
});

const info = await store.putBlob(
  { name: "classifier/v2.3", description: "Text classifier, trained 2026-01" },
  await readFile("./model-v2.3.bin"),
);
console.log(`Stored ${info.name}: ${info.size} bytes in ${info.chunks} chunks`);
console.log(`Digest: ${info.digest}`);

const data = await store.getBlob("classifier/v2.3"); // Uint8Array, or null

for (const obj of await store.list()) {
  console.log(`${obj.name}: ${obj.size} bytes`);
}
Enter fullscreen mode Exit fullscreen mode

getBlob() loads the whole object into memory. For large files, use store.get(), which returns a readable stream.

From the CLI:

nats object put models ./model-v2.3.bin --name "classifier/v2.3"
nats object get models "classifier/v2.3" --output ./downloaded-model.bin
nats object ls models
Enter fullscreen mode Exit fullscreen mode

For AI workloads: store model configs, LoRA adapters, and smaller model files in the Object Store. Keep the current version in a KV key such as models.classifier.current. Inference servers watch that key and pull the new object when it changes. That gives you a basic model registry without S3. For very large weights, measure first; dedicated object storage may still be the better fit.

Security and multi-tenancy

NATS security is built on accounts, users, and permissions, and it supports decentralized JWT auth for multi-tenant systems.

Users and permissions 

# nats-server.conf
authorization {
  users = [
    {
      user: "order-service"
      password: "$2a$11$..."
      permissions: {
        publish: ["orders.>"]
        subscribe: ["orders.*.status"]
      }
    },
    {
      user: "analytics-service"
      password: "$2a$11$..."
      permissions: {
        publish: { deny: [">"] }    # read-only
        subscribe: ["orders.>"]
      }
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

These permissions cover Core NATS subjects. A service that uses JetStream also needs access to the JetStream API subjects ($JS.API.>, or a narrower set), and a service that sends requests needs to subscribe to its reply inbox (_INBOX.>).

Accounts

Accounts give each tenant a separate namespace:

jetstream: enabled

accounts {
  TEAM_A: {
    users: [{ user: team_a_admin, password: "..." }]
    jetstream: enabled
  }
  TEAM_B: {
    users: [{ user: team_b_admin, password: "..." }]
    jetstream: enabled
  }
}
Enter fullscreen mode Exit fullscreen mode

Team A's orders stream and Team B's orders stream are separate, even on the same cluster. Accounts share data only through explicit exports and imports.

In production, use JWT-based auth managed with the nsc tool. Account and user credentials are signed keys, so the server doesn't need a central user database.

Putting it together for AI

Here's how the pieces fit in an AI agent system:

JetStream streams    agent.*.actions, inference.results
KV buckets           sessions, agent-state, config
Object Store         models, embeddings, documents
Request-reply        inference.chat.query  <->  model servers
Pub/sub              events.*  ->  any number of subscribers
Enter fullscreen mode Exit fullscreen mode

One agent turn:

  1. The user's message is published to chat.<session-id>.input.
  2. The agent loads its state from KV: agent-state key <session-id>.
  3. The agent sends a request to inference.chat.query and waits for the reply.
  4. The agent logs its reasoning to JetStream on agent.<session-id>.actions.
  5. The agent writes its updated context back to the same KV key.
  6. The agent loads tool config from the Object Store: tools/web-search/config.json.
  7. The response is published to chat.<session-id>.output.

All of this runs on one NATS cluster, with one set of runbooks and metrics.

Get started

  1. Install the server and the CLI: 
   brew install nats-server nats-io/nats-tools/nats
Enter fullscreen mode Exit fullscreen mode

Or download them from nats.io/download.

  1. Start the server with JetStream: 
   nats-server -js
Enter fullscreen mode Exit fullscreen mode
  1. Try it: 
   nats server info

   nats stream add TEST --subjects "test.*" --defaults
   nats pub test.hello "Hello NATS!"

   nats kv add config
   nats kv put config app.version "1.0.0"
   nats kv get config app.version
Enter fullscreen mode Exit fullscreen mode

Wrapping up

The point of this post: queuing, caching, and blob storage don't have to be three systems.

When you reach for Kafka for events, Redis for state, and S3 for files, check whether NATS covers the need first. Often it does, and you run one system instead of three.

NATS won't replace Postgres for transactional data. It can replace much of the infrastructure between your services, and that's where most of the operational work is.

Start with Core NATS, add JetStream when you need persistence, and see how far nats-server -js gets you before you add anything else.

Resources

What are you building with NATS? I'd like to hear about it.

Top comments (0)