DEV Community: Tarun Koshti

Beyond Caching: Scheduling Jobs with Redis, Node.js and BullMQ

Tarun Koshti — Fri, 09 Oct 2026 10:21:31 +0000

Hello Engineers, in this article we will see how Redis can be used as a job scheduler.
Most of us mainly use Redis to cache very frequently accessed data, but beyond data caching, Redis has lots of other applications as well. Some of them are listed below:

As a scheduler (topic of the current article)
As a cron service
As a message queue like SQS
For message delivery with publish/subscribe (Pub/Sub) in real-time applications

So in this article, we will see its job scheduler use case.
We'll build the examples with Node.js (TypeScript) and BullMQ, but the concepts apply to any language with a Redis job library.

Part 1: Why Redis?

1. Cost perspective

Say you're a small or medium-sized business whose application handles up to a million requests a day, and you already use Redis for caching. Adding a job queue costs you almost nothing: no new service to provision, no new bill, and no new system for your team to learn and monitor. At roughly 12 requests per second on average, your workload uses a tiny fraction of what a single Redis instance can handle. (Reusing your cache instance? See the configuration tip below.)

2. Precise and flexible timing

Jobs fire within milliseconds of their scheduled time, and delays can range from a few milliseconds to weeks. Many managed alternatives have limits here. AWS SQS, for example, caps message delays at 15 minutes.

3. Speed

Because Redis works in memory, scheduling and claiming a job typically takes under a millisecond. A single instance can handle tens of thousands of job operations per second, which is far more than most applications need.

4. Mature ecosystem

You rarely need to build a scheduler yourself. Battle-tested libraries exist for most languages: BullMQ for Node.js, Sidekiq for Ruby, Celery and RQ for Python, and Asynq for Go. They provide retries with backoff, attempt limits, priorities, rate limiting, repeatable jobs, and monitoring dashboards out of the box.

5. Simple mental model

The core pattern is easy to understand and debug: a sorted set where each job's score is derived from the time it should run. You can inspect pending jobs with a single command (ZRANGE bull:<queue>:delayed 0 -1 WITHSCORES), which makes troubleshooting straightforward compared to black-box managed services. (BullMQ stores the run time multiplied by 4096, so divide a score by 4096 to get the timestamp in milliseconds.)

6. One tool, many jobs

Job scheduling usually doesn't come alone. You often also need rate limiting (so you don't flood a customer's webhook endpoint), distributed locks, deduplication, and caching. Redis handles all of these, so a single piece of infrastructure covers several needs.

7. Portability

Redis runs anywhere: locally in Docker, on any cloud, or through managed services like AWS ElastiCache, Google Memorystore, and Azure Cache for Redis. Unlike cloud-specific options such as Google Cloud Tasks or AWS SQS, it doesn't lock you into one provider.

Configuration tip: memory policy

If you reuse your caching Redis for jobs (see point 1), check one setting first: caches and queues want opposite settings. A cache typically uses an eviction policy like allkeys-lru, which deletes old keys when memory fills up. A queue needs noeviction, or your jobs can silently disappear. So either run the queue on a separate Redis instance, or make sure the shared instance is configured for the queue's needs. (A separate database inside the same instance is not enough: the eviction policy applies to the whole instance, so cache traffic can still evict your jobs.)

What does that setting actually mean?

maxmemory-policy is the Redis setting that decides what happens when Redis reaches its memory limit (maxmemory). Options include allkeys-lru (delete the least recently used keys), volatile-lru (delete the least recently used keys that have a TTL), and others.
noeviction means Redis never deletes data to free memory. Once the limit is reached, it rejects new writes with an error (OOM command not allowed when used memory > 'maxmemory'), but keeps everything already stored. That's the safe behavior for a job queue, since your jobs won't vanish.
noeviction is Redis's own built-in default, so a fresh Redis (including the official Docker image) is already queue-safe. The risk comes when a shared instance is switched to allkeys-lru for caching.
Two things to keep in mind: keys that have a TTL still expire as normal, because expiry is not eviction. And by default maxmemory is 0 (no limit), so Redis keeps using memory until the machine runs out. In production, set maxmemory explicitly.

Does Redis work for large applications as well? Yes, of course, but you can't run Redis as a plain instance, because at large scale durability matters: a crash or restart without the right persistence settings can lose jobs.
At this scale, a bad configuration can lose all the scheduled jobs, causing permanent data loss.
Check the "Can I use it for very large applications?" section in Part 4 for more details.

Part 2: Code Example

Now it's time to see Redis in action.
In this demo, we'll use BullMQ (a Node.js job queue library built on Redis) to schedule a job with a delay and process it later.

Workspace setup

Let's set up the workspace first, so you can run every example yourself.

Tested with Node.js 22, BullMQ 6.3, ioredis 6.0, TypeScript 7.0 and Redis 7.

1. Install Node.js

Install Node.js 22 or later (a current LTS release) from nodejs.org, then check it:

node --version

2. Create the project and install the packages

mkdir bullmq-demo
cd bullmq-demo
npm init -y
npm install bullmq ioredis
npm install -D typescript tsx @types/node

bullmq — the job queue library.
ioredis — the Redis client BullMQ uses. Since BullMQ 6 it is an optional dependency, so you must install it yourself. Without it, the worker crashes on start with BullMQ could not load the optional 'ioredis' package.
typescript, @types/node — the TypeScript compiler and Node.js types.
tsx — runs .ts files directly, without a build step.

3. Add tsconfig.json

Create tsconfig.json in the project root:

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "nodenext",
    "moduleResolution": "nodenext",
    "types": ["node"],
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "noEmit": true
  },
  "include": ["src"]
}

"types": ["node"] is required since TypeScript 6, which no longer loads @types/node automatically. Without it you get errors like Cannot find name 'process'.
"noEmit": true, because tsx runs the code; tsc is only used to type-check:

npx tsc --noEmit

4. Start Redis

BullMQ needs a running Redis server. Pick one of the two options below.

Option A — plain Docker command

# start Redis in the background on port 6379
docker run -d --name bullmq-redis -p 6379:6379 redis:7-alpine

# check it is up (should print PONG)
docker exec bullmq-redis redis-cli ping

# when you are done: stop and remove the container
docker stop bullmq-redis
docker rm bullmq-redis

Option B — Docker Compose

Create docker-compose.yaml in the project root:

services:
  redis:
    image: redis:7-alpine
    # Turn on the append-only file, so scheduled jobs survive a restart
    # (the image's default is periodic snapshots only; see Part 4)
    command: redis-server --appendonly yes
    ports:
      - '6379:6379'
    volumes:
      - redis-data:/data

volumes:
  redis-data:

Then:

# start Redis in the background
docker compose up -d

# check it is up (should print PONG)
docker compose exec redis redis-cli ping

# when you are done: stop it (add -v to also delete the stored data)
docker compose down

Port 6379 already in use? Another Redis is probably running on your machine. Stop it, or map a different host port (-p 6380:6379, or "6380:6379" in the compose file) and set port: 6380 in connection.ts.

The redis:7-alpine image uses maxmemory-policy noeviction by default, which is the setting a job queue needs (see the configuration tip in Part 1).

maxmemory-policy is the Redis setting that decides what happens when Redis reaches its memory limit (maxmemory). Options include allkeys-lru (delete the least recently used keys), volatile-lru (delete the least recently used keys that have a TTL), and others.

noeviction means Redis never deletes data to free memory. Once the limit is reached, it rejects new writes with an error (OOM command not allowed when used memory > 'maxmemory'), but keeps everything already stored.

So if you run that image without changing anything, Redis will never delete your keys to free memory. That's the safe behavior for a job queue, since your jobs won't vanish. (noeviction is actually Redis's own built-in default; the image just doesn't override it.)

Two things to keep in mind: keys that have a TTL still expire as normal, because expiry is not eviction. And by default maxmemory is 0 (no limit), so Redis keeps using memory until the container or machine runs out. In production, set maxmemory explicitly, so you get the clear OOM error above instead of the process being killed.

5. Project structure

By the end of this part, your project will look like this:

bullmq-demo/
  docker-compose.yaml     (only if you use Option B)
  package.json
  tsconfig.json
  src/
    connection.ts
    worker.ts
    scheduler.ts
    workerManualRetry.ts

The code

First, connection.ts. It contains the shared connection and the job data interface required for this demo.

// src/connection.ts
// Shared Redis connection for the article demo
export const connection = {
  host: '127.0.0.1',
  port: 6379,
};

export const QUEUE_NAME = 'demo-reminders';

// Shape of the job body (what the scheduler passes as `data`).
// The retry settings travel WITH the job, so the manual retry worker
// (workerManualRetry.ts) can decide on its own whether to reschedule and how
// long to wait.
export interface ReminderJobData {
  orderId: string;
  reason: string;
  attempt: number; // which attempt this is, 1-based
  maxAttempts: number; // how many attempts to make in total before giving up
  retryDelayMs: number; // how long to wait before the next attempt
}

Now worker.ts.
The worker.ts is basically a long-running program: it keeps listening to the queue, and processes each job once it becomes ready (here, after its delay has elapsed).

// src/worker.ts
import { Worker } from 'bullmq';
import { connection, QUEUE_NAME } from './connection';

// The WORKER is the consumer side of the queue: it pulls jobs from Redis and processes them.
// Basically the worker contains the business logic to process the job's data, like sending emails, calling APIs and the database
// It does not reschedule or mark the job itself: returning normally marks the job completed, and throwing marks it failed — BullMQ then retries it according to the job's attempts/backoff options
// Keep it running in one terminal, then schedule jobs from another.
// The worker continuously pulls jobs for processing
//
// A Worker is bound to exactly ONE queue. If you have several queues you need to create a separate worker instance for every queue name
// The first argument is the queue name (Example: "DEMO_QUEUE")
// The second argument is the async function which contains the business logic
// The third argument is the connection configuration
const worker = new Worker(
  QUEUE_NAME,
  async (job) => {
    // job.id   — unique id of this job in the queue
    // job.name — the job's name (set by queue.add('reminder', ...) in
    //            scheduler.ts); use it to route different job types handled
    //            by the same worker
    // job.data — the payload passed when the job was scheduled
    console.log('Processing queue:', QUEUE_NAME);
    console.log(`[${new Date().toLocaleTimeString()}] processing job ${job.id}:`, job.data);
  },

  { connection },
);

// Lifecycle events. The worker keeps a long-lived connection to Redis and emits
// an event for every job it finishes.
worker.on('completed', (job) => console.log(`  done: ${job.id}`)); // job succeeded
worker.on('failed', (job, err) => console.error(`  failed: ${job?.id} — ${err.message}`)); // job threw

// Redis connection problems are reported here. Without this listener, BullMQ
// prints a raw stack trace on every reconnect attempt.
worker.on('error', (err) => console.error('  worker error:', err.message));

Now scheduler.ts.
The scheduler is used to add jobs to the queue (and to cancel them before they run).
Run it to add a job to the queue.

// src/scheduler.ts
import { Queue } from 'bullmq';
import { connection, QUEUE_NAME } from './connection';

// A Queue is the producer-side handle to a queue in Redis: you use it to ADD
// jobs and to look them up / remove them. It does not process anything — that's
// the worker's job.
//   arg 1 — queue name (must match the worker's queue name)
//   arg 2 — options, including the Redis connection
const queue = new Queue(QUEUE_NAME, { connection });

// Redis connection problems are reported here. Without this listener, BullMQ
// prints a raw stack trace on every reconnect attempt.
queue.on('error', (err) => console.error('queue error:', err.message));

// The SCHEDULER enqueues a delayed job, or cancels one before it runs.
//   npx tsx src/scheduler.ts            -> schedule a job (fires in 5s)
//   npx tsx src/scheduler.ts cancel <id> -> cancel that job
async function main() {
  // Read CLI args: e.g. `... scheduler.ts cancel 42` -> action="cancel", jobId="42"
  const [action, jobId] = process.argv.slice(2);

  if (action === 'cancel') {
    // getJob(id) looks a job up by id. Returns the Job, or undefined if it has
    // already run or was removed.
    const job = await queue.getJob(jobId);
    if (job) {
      // remove() deletes the job from the queue. For a delayed job that hasn't
      // run yet, this cancels it — the worker will never see it.
      await job.remove();
      console.log(`cancelled job ${jobId} — it will not run`);
    } else {
      console.log(`no job ${jobId} found (already ran or removed)`);
    }
  } else {
    const delayMs = 5000;

    // add() puts a new job on the queue.
    //   arg 1 — job name ('reminder'): a label the worker can switch on to
    //           handle different job types on the same queue.
    //   arg 2 — data: the payload, must be JSON-serializable. The worker reads
    //           this as job.data. The `attempt` / `maxAttempts` / `retryDelayMs`
    //           fields are only used by the manual retry worker
    //           (workerManualRetry.ts); the basic worker just logs them.
    //   arg 3 — options: here `delay` (ms) holds the job back so it only becomes
    //           available to the worker after that time. Other options live here
    //           too: `attempts`, `backoff`, `jobId`, `priority`, etc. (Recurring
    //           jobs are not an add() option — use queue.upsertJobScheduler().)
    const job = await queue.add(
      'reminder',
      {
        orderId: 'order-123',
        reason: 'unpaid',
        attempt: 1,
        maxAttempts: 3,
        retryDelayMs: 3000,
      },
      { delay: delayMs },
    );
    console.log(`scheduled job ${job.id} to run in ${delayMs / 1000}s`);
    console.log(`cancel it with: npx tsx src/scheduler.ts cancel ${job.id}`);
  }

  // Close the connection so this short-lived script can exit (the worker, by
  // contrast, stays open to keep processing).
  await queue.close();
}

main();

In the above example, you can see how Redis can be used for simple job scheduling.

Now, what if your job fails during processing?
You need to retry it. In the next example, I'll show you how to retry a failed job.

Manual job retry example

Here the job body carries the retry settings, and the worker reschedules the job itself when the work fails.

// src/workerManualRetry.ts
//
// MANUAL RETRY EXAMPLE — an alternative to worker.ts.
//
// Instead of letting BullMQ retry (via `attempts` + `backoff` in the job
// options), the retry settings travel in the JOB BODY and this worker decides
// for itself whether to try again. On failure it adds a NEW job to the same
// queue with attempt + 1, delayed by retryDelayMs.
//
// Run EITHER this worker OR worker.ts — not both. They consume the same queue,
// so if both run they will compete for the same jobs.
//
// Trade-offs vs BullMQ's built-in retry:
//   * the retry is a NEW job with a NEW id (built-in retry reuses the same job
//     id and increments job.attemptsMade)
//   * because we CATCH the error, BullMQ never sees a failure — every attempt
//     is recorded as 'completed' and the 'failed' event never fires
// Use this only when you need that control; otherwise prefer `attempts` + `backoff`.
import { Queue, Worker } from 'bullmq';
import { connection, QUEUE_NAME, ReminderJobData } from './connection';

// A Queue handle, so the worker can put the NEXT attempt back on the same queue.
// Only needed because we are retrying manually.
const queue = new Queue(QUEUE_NAME, { connection });
queue.on('error', (err) => console.error('  queue error:', err.message));

// Simulated work — this is the part that can fail in real life (SMTP down,
// payment API timing out, etc.). It fails on attempt 1 and succeeds from
// attempt 2, so the retry path is easy to watch in the demo.
async function sendReminder(data: ReminderJobData): Promise<void> {
  if (data.attempt === 1) {
    throw new Error('payment provider timed out');
  }
  console.log(`  reminder sent for ${data.orderId} (${data.reason})`);
}

const worker = new Worker<ReminderJobData>(
  QUEUE_NAME,

  async (job) => {
    // The retry settings come from the job body, not the job options.
    const { attempt, maxAttempts, retryDelayMs } = job.data;

    console.log('Processing queue:', QUEUE_NAME);
    console.log(
      `[${new Date().toLocaleTimeString()}] job ${job.id} — attempt ${attempt}/${maxAttempts}`,
    );

    try {
      await sendReminder(job.data);
      console.log(`  success on attempt ${attempt}`);
    } catch (err) {
      const reason = (err as Error).message;

      if (attempt < maxAttempts) {
        // RESCHEDULE: add a NEW job to the same queue carrying an incremented
        // attempt counter, delayed by retryDelayMs. That delayed job becomes
        // "attempt 2", and this one finishes normally.
        //   arg 1 — same job name, so the same worker logic handles it
        //   arg 2 — the same body, with attempt bumped by 1
        //   arg 3 — delay: wait this long before the retry becomes available
        await queue.add(job.name, { ...job.data, attempt: attempt + 1 }, { delay: retryDelayMs });

        console.log(
          `  attempt ${attempt} failed (${reason}) — retrying in ${retryDelayMs / 1000}s as attempt ${attempt + 1}`,
        );
      } else {
        // No attempts left: stop here instead of scheduling another job.
        console.error(`  attempt ${attempt} failed (${reason}) — no attempts left, giving up`);
      }
    }
  },

  { connection },
);

// NOTE: 'completed' fires for every attempt here — including the ones whose
// work failed — because the error was caught above. 'failed' only fires for
// errors we did not catch.
worker.on('completed', (job) => console.log(`  done: ${job.id}`));
worker.on('failed', (job, err) => console.error(`  failed: ${job?.id} — ${err.message}`));

// Redis connection problems are reported here. Without these listeners, BullMQ
// prints a raw stack trace on every reconnect attempt.
worker.on('error', (err) => console.error('  worker error:', err.message));

Part 3: From demo to a real app

In Part 2, we scheduled a job, cancelled it, and retried a failed job manually.
In this part, we'll see:

The easier, recommended way to retry: BullMQ's built-in retry.
A real example from an Express app: unpaid order reminders.
A few tips before you take it to production.

Built-in retry (the recommended way)

With the manual retry, the worker caught the error and added a new job by itself.
BullMQ can do this for you: set attempts and backoff when you add the job, and simply throw from the worker when the work fails. BullMQ then retries the same job automatically.

In src/scheduler.ts, add attempts and backoff to the job options:

const job = await queue.add(
  'reminder',
  {
    orderId: 'order-123',
    reason: 'unpaid',
    attempt: 1,
    maxAttempts: 3,
    retryDelayMs: 3000,
  },
  { delay: delayMs, attempts: 3, backoff: { type: 'exponential', delay: 1000 } },
);

attempts: 3 — try the job at most 3 times in total (the first try + 2 retries).
backoff — how long to wait between tries. exponential with delay: 1000 waits 1s before the 2nd try and 2s before the 3rd (the wait doubles each time). Use type: 'fixed' to wait the same time every try.

The attempt, maxAttempts and retryDelayMs fields in the body are only used by the manual retry worker; the built-in retry ignores them.

Now the worker. Notice there is no try/catch and no re-adding of the job:

// src/workerBuiltInRetry.ts
//
// BUILT-IN RETRY EXAMPLE — the recommended way.
//
// The worker simply THROWS when the work fails. BullMQ sees the failure and
// retries the SAME job automatically, using the `attempts` and `backoff`
// options given when the job was added (see scheduler.ts).
//
// Run EITHER this worker OR worker.ts / workerManualRetry.ts — not more than
// one. They consume the same queue, so they would compete for the same jobs.
import { Worker } from 'bullmq';
import { connection, QUEUE_NAME } from './connection';

// Simulated work — fails on the first attempt and succeeds on the second, so
// the retry is easy to watch.
async function sendReminder(orderId: string, attempt: number): Promise<void> {
  if (attempt === 1) {
    throw new Error('payment provider timed out');
  }
  console.log(`  reminder sent for ${orderId}`);
}

const worker = new Worker(
  QUEUE_NAME,

  async (job) => {
    // job.attemptsMade — how many attempts have already FAILED (0 on the first run)
    // job.opts.attempts — the total attempts allowed, set when the job was added
    const attempt = job.attemptsMade + 1;
    console.log(
      `[${new Date().toLocaleTimeString()}] job ${job.id} — attempt ${attempt}/${job.opts.attempts}`,
    );

    // No try/catch here: if this throws, BullMQ marks the attempt as failed,
    // waits according to `backoff`, and runs the same job again.
    await sendReminder(job.data.orderId, attempt);
  },

  { connection },
);

// 'failed' fires for every failed attempt; 'completed' fires once, when an
// attempt finally succeeds.
worker.on('completed', (job) => console.log(`  done: ${job.id}`));
worker.on('failed', (job, err) =>
  console.error(`  failed: ${job?.id} (attempt ${job?.attemptsMade}) — ${err.message}`),
);

// Redis connection problems are reported here. Without this listener, BullMQ
// prints a raw stack trace on every reconnect attempt.
worker.on('error', (err) => console.error('  worker error:', err.message));

Run the worker in one terminal and the scheduler in another:

npx tsx src/workerBuiltInRetry.ts

npx tsx src/scheduler.ts

Output:

[8:06:41 AM] job 1 — attempt 1/3
  failed: 1 (attempt 1) — payment provider timed out
[8:06:42 AM] job 1 — attempt 2/3
  reminder sent for order-123
  done: 1

The first attempt failed, BullMQ waited 1 second, and ran job 1 again, which succeeded.

Manual retry vs built-in retry

	Manual retry (Part 2)	Built-in retry
Retry settings live in	the job body	the job options
On failure, the worker	catches the error and adds a new job	just throws
Job id on retry	a new id every time	the same id
`failed` event	never fires (every attempt is "completed")	fires for every failed attempt
Code you write	try/catch + re-add logic	nothing extra

Use the built-in retry by default. Reach for the manual retry only when you need full control over each retry, for example a different payload for every attempt.

Real example: unpaid order reminders

Now let's see how this looks in a real app. The example is an Express + MongoDB app with this rule:

When an order is created, remind the customer every 30 seconds, up to 3 times, while the order is unpaid. When the order is paid, cancel the remaining reminders.

The full source code is on GitHub (link at the end). Here are the key parts.

1. Two different "3"s in the config

// src/constants/appConstants.ts
export const QUEUE_CONFIG: Record<QueueName, QueueSettings> = {
  [QUEUE_NAMES.ORDER_REMINDERS]: {
    jobName: 'unpaid-reminder',
    delay: 30_000,
    maxNotifications: 3,
    attempts: 3,
    backoff: { type: 'exponential', delay: 1_000 },
    removeOnComplete: { age: 24 * 60 * 60 }, // seconds
    removeOnFail: { age: 7 * 24 * 60 * 60 },
  },
};

These two numbers look the same, but they mean different things:

maxNotifications is business logic: how many reminders to send while the order is unpaid.
attempts + backoff is BullMQ's built-in retry: how many times to retry one reminder if sending it fails.

The last two lines tell BullMQ when it may delete finished jobs from Redis: completed ones after 24 hours, failed ones after 7 days. By default, BullMQ keeps them forever (more on this in Part 4).

2. Scheduling a reminder with a deterministic job id

// src/services/orderReminders.ts
function reminderJobId(orderId: string, notificationNo: number): string {
  return `reminder:${orderId}:${notificationNo}`;
}

export async function scheduleReminder(orderId: string, notificationNo: number): Promise<string> {
  const jobId = reminderJobId(orderId, notificationNo);
  await addJob<ReminderJobData>(QUEUE, { orderId, notificationNo }, { jobId });
  await OrderModel.findByIdAndUpdate(orderId, { reminderJobId: jobId });
  return jobId;
}

Instead of letting BullMQ generate a random job id, we build one from the order id and the reminder number, for example reminder:6aaf...:2. Adding a job with an id that already exists does nothing, so the same reminder can never be scheduled twice. We also store the id on the order, so we can find and cancel the job later.

3. The worker: check, schedule the next reminder, then send

// src/services/orderReminders.ts
export async function processReminder(job: Job<ReminderJobData>): Promise<void> {
  const { orderId, notificationNo } = job.data;
  const order = await OrderModel.findById(orderId);

  // Stop if the order is gone or already paid.
  if (!order || order.status === 'paid') return;

  // Schedule the next reminder BEFORE sending this one.
  if (notificationNo < maxNotifications) {
    await scheduleReminder(orderId, notificationNo + 1);
  } else {
    await OrderModel.findByIdAndUpdate(orderId, { reminderJobId: null });
  }

  // If this throws, BullMQ retries it per `attempts` + `backoff`.
  await sendReminder(order.id, notificationNo);
}

Three things are happening here:

The check comes first. If the order was paid in the meantime, the job does nothing.
Each reminder schedules the next one, until maxNotifications is reached. This is how "remind up to 3 times" works: a chain of delayed jobs.
The next reminder is scheduled before sending. If sending fails on every retry, the next reminder is already in the queue, so one failed reminder doesn't stop the rest. And because the job id is deterministic, when BullMQ retries this job, scheduling the next reminder again is a no-op.

4. Cancelling on payment

// src/controllers/orderController.ts (payOrder)
// Mark paid first, so any reminder racing this request sees `paid` and no-ops.
order.status = 'paid';
await order.save();

// Then remove the one pending reminder — this ends the chain permanently.
// If Redis is unreachable this fails, but the payment is already saved and
// the worker's paid-status check stops the reminder when it runs, so the
// payment still succeeds.
try {
  await cancelReminder(order.id);
  order.reminderJobId = null;
} catch (err) {
  console.error('Could not cancel reminder (it will no-op when it runs):', (err as Error).message);
}

The order is marked paid first, then the pending reminder job is removed. Since each reminder schedules the next one, removing the single pending job ends the whole chain.

Cancelling is best-effort: if Redis happens to be down at that moment, the payment still succeeds. The reminder job stays in the queue, but when it runs, the check at the top of the worker sees the order is paid and does nothing.

The result, with an unpaid order and a paid one:

Reminder 1/3: order 6aaf3f79ecf4cf995f6ab3ed is still unpaid
Reminder 2/3: order 6aaf3f79ecf4cf995f6ab3ed is still unpaid
Reminder 3/3: order 6aaf3f79ecf4cf995f6ab3ed is still unpaid

The unpaid order got exactly 3 reminders; the paid order got none.

Before you go to production

1. Handle Redis outages. With the default settings, while Redis is down, every queue command waits in memory until Redis comes back, so your API requests just hang. The tempting fix is to cap the reconnect attempts so the client gives up, but then it never reconnects: after the outage, the app stays broken until you restart it. The right setup has three parts.

Keep reconnecting forever, with a growing (but capped) wait between attempts:

// src/config/config.ts
redis: {
  host: process.env.REDIS_HOST ?? '127.0.0.1',
  port: Number(process.env.REDIS_PORT ?? 6379),
  // Required by BullMQ for its blocking worker connection.
  maxRetriesPerRequest: null,
  // Keep reconnecting forever, waiting a little longer after each failed
  // attempt (200ms, 400ms, ...) up to REDIS_RETRY_DELAY_MS.
  retryStrategy: (times: number): number => Math.min(times * 200, redisMaxRetryDelayMs),
},

Make the queue (the producer) fail fast while Redis is down, instead of waiting:

// src/services/queue.ts
// enableOfflineQueue: false — while Redis is down, adding a job fails
// immediately instead of waiting in memory until Redis comes back, so the
// API can answer the client right away. (Workers keep the default.)
queue = new Queue(name, { connection: { ...config.redis, enableOfflineQueue: false } });

And at startup, wait for Redis only for a limited time, so the app exits instead of accepting requests it cannot serve:

// src/services/queue.ts
export async function verifyRedisConnection(timeoutMs: number): Promise<void> {
  let timer: NodeJS.Timeout | undefined;
  const timeout = new Promise<never>((_, reject) => {
    timer = setTimeout(
      () => reject(new Error(`Redis not reachable within ${timeoutMs}ms`)),
      timeoutMs,
    );
  });
  try {
    await Promise.race([getQueue(QUEUE_NAMES.ORDER_REMINDERS).waitUntilReady(), timeout]);
  } finally {
    clearTimeout(timer);
  }
}

With this, while Redis is down, requests fail in milliseconds; when it comes back, the API and the worker recover by themselves.

2. Don't leave half-done work behind. Creating an order is two steps: save the order, then schedule its reminder. If the second step fails (for example, Redis is down), remove the order and tell the client to retry. Otherwise the client sees an error, but an order that will never be reminded stays in your database:

// src/controllers/orderController.ts (createOrder)
const order = await OrderModel.create({ item, amount });

try {
  const jobId = await scheduleReminder(order.id, 1);
  order.reminderJobId = jobId;
} catch (err) {
  // The reminder could not be scheduled (e.g. Redis is down). Don't keep an
  // order that would never be reminded: remove it and tell the client to
  // retry, so an error response never leaves a hidden order behind.
  await OrderModel.deleteOne({ _id: order._id });
  console.error('Could not schedule reminder:', (err as Error).message);
  throw new AppError('could not schedule reminder, please retry', 503);
}

3. Add error listeners to every Queue and Worker. Without them, BullMQ prints a raw stack trace on every reconnect attempt while Redis is down, which floods your logs and hides everything else:

queue.on('error', (err) => console.error(`Queue ${name} connection error:`, err.message));
reminderWorker.on('error', (err) =>
  console.error('Reminder worker connection error:', err.message),
);

4. Run workers as a separate process. In the example app, the worker runs inside the API process to keep things simple. In production, run workers as their own process, so you can scale them independently and a slow job never blocks your API.

5. One worker per queue. A Worker processes exactly one queue. If you have several queues, create one Worker for each.

Full source code

The complete app (Express + MongoDB + BullMQ) is on GitHub: tarun-koshti/Redis-BullMq

To run it:

git clone https://github.com/tarun-koshti/Redis-BullMq.git
cd Redis-BullMq
docker compose up -d
npm install
cp .env.example .env
npm run dev

In the next part, we'll see what changes when you run Redis-backed jobs at a very large scale.

Part 4: Can I use it for very large applications?

Yes. Redis-backed job queues run in production at very large scale. But the approach changes as you grow. Small apps can rely on defaults; large apps need to plan capacity, partitioning, and durability deliberately.

1. Start by measuring jobs, not requests

Your request count and your job count are different numbers. An app handling 10,000 requests per second might create only a few hundred jobs per second if jobs come from failures, like webhook retries. Estimate your actual job rate first, and remember that each job costs many Redis operations over its lifecycle (adding, moving from delayed to ready, claiming, completing, plus lock renewals for long-running jobs). In our test with BullMQ 6, a single delayed job ran about 40 Redis commands, packed into just 2 Lua script calls.

2. Calculate your memory budget

A simple formula estimates how many jobs sit in Redis at once: job rate × average delay. At 10,000 jobs per second with a 15-minute delay, that's 9 million pending jobs. In our test, a job with a 1 KB payload took about 1.9 KB in Redis, so 9 million of them need roughly 17 GB of RAM just for the backlog.

Finished jobs count too: by default, BullMQ keeps every completed and failed job in Redis. Set removeOnComplete and removeOnFail so they don't pile up:

await queue.add('reminder', data, {
  delay: 30_000,
  removeOnComplete: 1000, // keep only the last 1,000 completed jobs
  removeOnFail: 5000, // keep more failed jobs, for debugging
});

3. Keep job payloads small

Store only a job ID and a few essential fields in Redis, and keep the full payload in your primary database or object storage. In our test, a job carrying just an order ID and a number took about 0.44 KB, over 4 times less than the 1 KB payload above. The bigger your payloads, the bigger the saving.

4. Scale horizontally by partitioning queues

Redis executes commands on a single main thread, so a bigger machine only helps so much. Instead, split work across multiple queues, for example by customer ID or event type, and spread them across instances or a Redis Cluster. Because queue libraries use multi-key operations, each individual queue must stay on a single shard. In BullMQ, you do this with a hash tag in the queue prefix (and the worker must use the same prefix):

// All keys of this queue share the {webhooks:shard3} hash tag,
// so Redis Cluster keeps the whole queue on one shard.
const queue = new Queue('webhooks', { connection, prefix: '{webhooks:shard3}' });

Many smaller queues scale; one giant queue doesn't.

5. Scale workers independently

Workers are stateless and can be added or removed freely. Watch the load they put on Redis, though: hundreds of workers polling aggressively create significant traffic on their own. Mature libraries avoid this with blocking commands and smarter polling, which is one more reason to prefer them over a hand-rolled scheduler at scale.

6. Plan for backlog spikes

The biggest danger at scale is not normal traffic but an outage somewhere else. If a major customer's webhook endpoint goes down, millions of retries can pile up in memory within hours. Protect yourself with per-destination rate limits, circuit breakers that pause delivery to failing endpoints, the noeviction memory policy, and alerts on memory usage and queue depth.

7. Strengthen durability to match the stakes

At large scale, even rare job loss affects many users. Start by checking persistence: the official Redis Docker image has the append-only file (AOF) turned off, and only saves snapshots now and then (after 1 hour if at least 1 key changed, after 5 minutes if 100 changed, after 1 minute if 10,000 changed). So a crash can lose several minutes of jobs, or up to an hour on a quiet instance. Turn on the AOF:

# redis.conf
appendonly yes
appendfsync everysec   # write to disk every second: a crash loses at most ~1 second

With Docker, the same is redis-server --appendonly yes as the container command.

Beyond that, options include WAITAOF (Redis 7.2+) to confirm writes are persisted before acknowledging them, durable managed services such as AWS MemoryDB, or keeping your primary database as the source of truth, with a periodic sweep to catch any jobs Redis lost.

8. Consider a hybrid architecture

Many large systems combine tools, each doing what it does best. A durable log like Kafka or a queue like SQS handles ingestion, a database stores job state, and Redis provides fast, precise timing. This gets you Redis's speed without depending on it alone for durability.

9. Invest in monitoring

At scale, you need visibility into queue depth, scheduling lag (how late jobs run compared to their scheduled time), memory usage, failure and dead-letter rates, and Redis operations per second. Scheduling lag is especially useful: it rises early when workers or Redis start falling behind.

Redis doesn't stop working at large scale, but it stops forgiving mistakes. With partitioned queues, small payloads, the right durability settings, and good monitoring, it can serve as the scheduling backbone of very large applications.

Wrapping up

In this article, we saw:

why Redis is a good fit for scheduling jobs (Part 1),
how to schedule, cancel and retry jobs with BullMQ (Part 2),
how this looks in a real Express app (Part 3),
and what changes at a very large scale (Part 4).

If you already run Redis, you can start with a single queue and grow from there.

The complete source code is on GitHub: tarun-koshti/Redis-BullMq

In the next article, we'll see how to use Redis as a cron service, to run jobs on a repeating schedule, like every night at 2 AM.

Thanks for reading!

AsyncLocalStorage in Node.js: stop passing requestId to every function

Tarun Koshti — Sun, 30 Aug 2026 07:31:48 +0000

Hello folks,

Recently I used a really interesting Node.js feature: AsyncLocalStorage.

What is AsyncLocalStorage?

AsyncLocalStorage lets you pass data across functions without passing it through function arguments. It ships with Node itself, in node:async_hooks, so there is nothing to install.

But wait — we can do the same thing with a global variable! So why do we need this?

Well, yes, you can do it with a global variable. But what if your app is serving 50 concurrent users? Let's say you store the request id in one:

User 1 arrives → gets request id U1 → their request takes 1 second to reach the next step
User 2 arrives in the meantime → a new request id is generated → assigned to that same global variable → now the rest of user 1's logs point at somebody else's request
Make that 50 concurrent users → 50 new request ids → tracing a single request becomes impossible

Global variables are fine for project-wide constants, but not for request-specific data.

Then what's the problem with passing it through function arguments?

Let's say you want a unique request identifier in your logs. You have 20 functions in the chain, and the only place that actually needs that identifier is the logger.

Most of those functions never log anything themselves — only the ones that make the HTTP call and write to the database do. But it is still your obligation to pass the request id through every downstream function, just so it can reach the one place that needs it.

That is where AsyncLocalStorage comes in. It carries the request id in its context, and whoever needs it simply reads it from the store.

Let's understand this with an example.

Example 1: passing requestId through every function argument

Notice how the request id has to be passed to every function on the way down.

// The "before" picture: requestId is threaded through every function argument.
// Count how many of these functions accept requestId without ever reading it.

// Shape of the JSON-ish payloads we pass around.
type TObject = Record<string, string | number | object | boolean>;

// Logger function.
// This is the ONLY function that actually reads requestId. Every other
// signature below just carries it along so that it can reach this one.
async function logger(logData: TObject, requestId: string) {
  const logObject = {
    requestId,
    request: logData,
  };
  console.log(JSON.stringify(logObject));
}

// Function to handle the request.
// The request lands here first, and this is where requestId is born.
async function handleRequest(requestData: TObject) {
  const requestId = crypto.randomUUID();

  // Example function to make an HTTP call
  const apiResult = await apiCall(requestData, requestId);

  // Example function to save the data in the database
  await saveData(apiResult, requestId);
}

// Function to make the HTTP request.
// requestId is dead weight here: it is forwarded to logger(), nothing more.
async function apiCall(requestData: TObject, requestId: string) {
  // Stand-in for a real API call: wait instead of hitting the network.
  await new Promise((resolve) => setTimeout(resolve, 200));

  // Dummy API response
  const response: TObject = {
    success: true,
    data: {
      userId: 'U1',
      paymentId: 'U_P1',
      sentPayload: requestData,
    },
  };

  await logger(response, requestId);
  return response;
}

// Function to save the data in the database.
// Same story: requestId is pure pass-through.
async function saveData(apiResult: TObject, requestId: string) {
  // Stand-in for a real database write.
  await new Promise((resolve) => setTimeout(resolve, 100));

  await logger({ saved: true, record: apiResult }, requestId);
}

// Run it:  npx tsx example.ts
await handleRequest({ userId: 'U1', amount: 4999 });

Here, just to get the request id into the logger, we had to pass it through 2 functions. Now imagine 20 functions instead — every single one of them has to accept the request id and forward it, and the moment one of them forgets, that log line loses its correlation.

Example 2: the same code with AsyncLocalStorage

Now it's time to see the real magic: how the request id travels without being passed through function arguments.

import { AsyncLocalStorage } from 'node:async_hooks';

interface Context {
  requestId: string;
}

const asyncLocalStorage = new AsyncLocalStorage<Context>();

// Read the current request's id out of the store. Exposing this accessor
// instead of the store itself keeps run() in one place.
function getRequestId(): string | undefined {
  return asyncLocalStorage.getStore()?.requestId;
}

// Shape of the JSON-ish payloads we pass around.
type TObject = Record<string, string | number | object | boolean>;

// Logger function.
// Only this function reads requestId, and it pulls it from the store instead
// of receiving it as an argument. Every other signature below stays clean.
async function logger(logData: TObject) {
  const requestId = getRequestId();
  const logObject = {
    requestId,
    request: logData,
  };
  console.log(JSON.stringify(logObject));
}

// Function to handle the request.
async function handleRequest(requestData: TObject) {
  // Example function to make an HTTP call
  const apiResult = await apiCall(requestData);

  // Example function to save the data in the database
  await saveData(apiResult);
}

// Function to make the HTTP request.
// It never takes or mentions requestId, yet its log line still carries one.
async function apiCall(requestData: TObject) {
  // Stand-in for a real API call: wait instead of hitting the network.
  await new Promise((resolve) => setTimeout(resolve, 200));

  // Dummy API response
  const response: TObject = {
    success: true,
    data: {
      userId: 'U1',
      paymentId: 'U_P1',
      sentPayload: requestData,
    },
  };

  await logger(response);
  return response;
}

// Function to save the data in the database.
// Same story: a clean signature, and a correlated log line anyway.
async function saveData(apiResult: TObject) {
  // Stand-in for a real database write.
  await new Promise((resolve) => setTimeout(resolve, 100));

  await logger({ saved: true, record: apiResult });
}

// The requestId is created once, here at the boundary. Everything called
// inside run() can read it: across awaits, through nested calls, all the way
// down to logger().
asyncLocalStorage.run(
  {
    requestId: crypto.randomUUID(),
  },
  async () => {
    await handleRequest({ userId: 'u1', paymentId: 'U_P1' });
  },
);

Here you can see we don't pass anything through the functions. We store the requestId once in AsyncLocalStorage, and the logger picks it up at the final logging step.

Both log lines come out carrying the same id:

{"requestId":"84762d3d-5f73-428b-9995-29f9b34a477a","request":{"success":true,"data":{...}}}
{"requestId":"84762d3d-5f73-428b-9995-29f9b34a477a","request":{"saved":true,"record":{...}}}

Compare the two signatures side by side, and that is the whole argument:

before:  logger(logData, requestId)   apiCall(requestData, requestId)   saveData(apiResult, requestId)
after:   logger(logData)              apiCall(requestData)              saveData(apiResult)

Example 3: the same thing in Express, using express-http-context

Both examples above were plain scripts. In a real app the boundary is your HTTP layer, and there is a small package that wires exactly this up for you: express-http-context.

It is worth knowing what it actually is before you reach for it. It has no dependencies, and under the hood it is the very same AsyncLocalStorage we just used by hand — its middleware is literally asyncLocalStorage.run(new Map(), () => next()). So this is not an alternative approach, it is the same approach with the plumbing already written.

npm install express express-http-context

// The same example as before, now wired into Express with express-http-context.
//
// In the AsyncLocalStorage version we wrote three pieces of plumbing by hand:
//   1. the store     ->  new AsyncLocalStorage<Context>()
//   2. the boundary  ->  asyncLocalStorage.run({ requestId }, ...)
//   3. the accessor  ->  getRequestId()
//
// express-http-context takes the place of all three. It is a small wrapper
// with no dependencies, and under the hood it is the very same
// AsyncLocalStorage -- it just holds the store as a Map<string, unknown>, so
// you address values by key instead of through a typed Context object.

import express from 'express';
import httpContext from 'express-http-context';
import { randomUUID } from 'node:crypto';

const app = express();
app.use(express.json());

// 1. Create the per-request store.
//    This one line replaces both `new AsyncLocalStorage()` and the
//    `asyncLocalStorage.run(...)` wrapper from the previous example.
//    Internally the middleware is literally:
//        asyncLocalStorage.run(new Map(), () => next())
//    Because next() is called inside run(), every middleware, route and
//    function further down the chain inherits the same store.
app.use(httpContext.middleware);

// 2. Put the request id into that store, once, at the boundary.
//    Previously this was the object we handed to run({ requestId: ... }).
//    Careful: set() silently does nothing when the middleware above is
//    missing, so there is no error to tell you the context was never opened.
app.use((req, res, next) => {
  httpContext.set('requestId', req.header('x-request-id') ?? randomUUID());
  next();
});

// Logger function.
// As we saw above, only this function reads requestId, and it pulls it from
// the store instead of receiving it as an argument. Every other signature
// below stays clean.
// The one change from the previous example: our custom getRequestId() helper
// is gone, and httpContext.get('requestId') now does that job for us.
async function logger(logData) {
  const requestId = httpContext.get('requestId');
  const logObject = {
    requestId,
    request: logData,
  };
  console.log(JSON.stringify(logObject));
}

// Function to make the HTTP request.
// Unchanged from the AsyncLocalStorage example: it never takes or mentions
// requestId, yet its log line still carries one.
async function apiCall(requestData) {
  // Stand-in for a real API call: wait instead of hitting the network.
  await new Promise((resolve) => setTimeout(resolve, 200));

  // Dummy API response
  const response = {
    success: true,
    data: {
      userId: 'U1',
      paymentId: 'U_P1',
      sentPayload: requestData,
    },
  };

  // Log. Nothing about the request id is passed in.
  await logger(response);
  return response;
}

// Function to save the data in the database.
// Also unchanged: a clean signature, and a correlated log line anyway.
async function saveData(apiResult) {
  // Stand-in for a real database write.
  await new Promise((resolve) => setTimeout(resolve, 100));

  // Log again, still without passing the request id anywhere.
  await logger({ saved: true, record: apiResult });
}

// This route takes the place of handleRequest() from the previous example.
// Notice there is no run() call to wrap it in: httpContext.middleware already
// opened the context for this request, so everything called from here can
// read the id.
app.post('/orders', async (req, res) => {
  // Example function to make an HTTP call
  const apiResult = await apiCall(req.body);

  // Example function to save the data in the database
  await saveData(apiResult);

  // Reading the id here too, to hand it back to the caller -- handy in a real
  // app, since the client can then quote it in a bug report.
  res.status(201).json({ success: true, requestId: httpContext.get('requestId') });
});

// Fire two requests at the same time and watch the ids stay separate:
//   curl -X POST localhost:3000/orders -H 'content-type: application/json' -d '{"userId":"u1"}'
app.listen(3000, () => {
  console.log('listening on http://localhost:3000');
});

Run it with node, then fire two requests at once. The log lines interleave, but each keeps its own id:

{"requestId":"04757118-...","request":{"success":true,"data":{"sentPayload":{"userId":"u2",...}}}}
{"requestId":"cdce047b-...","request":{"success":true,"data":{"sentPayload":{"userId":"u1",...}}}}
{"requestId":"04757118-...","request":{"saved":true,"record":{...u2...}}}
{"requestId":"cdce047b-...","request":{"saved":true,"record":{...u1...}}}

Two things to watch out for with this package specifically:

set() fails silently if you forget app.use(httpContext.middleware), or if you call it outside a request. Nothing is stored and no error is raised.
The store is a Map<string, unknown>, so httpContext.get('requestId') gives you any. In a TypeScript codebase the hand-rolled AsyncLocalStorage<Context> version keeps its types, which is a good reason to stay with the built-in.

A few things to keep in mind

AsyncLocalStorage is not magic, and the context does not follow your data everywhere:

It does not survive a queue. Work you push onto a job queue during a request and run later has no context. Capture it with AsyncLocalStorage.snapshot() at enqueue time.
EventEmitter listeners inherit the context of whoever calls .emit(), not the one they were registered in. Wrap them with AsyncResource.bind().
A shared or memoised promise belongs to whoever created it. If two requests await the same in-flight promise, its body runs in the first request's context — which can silently mix up more than just logs.
Prefer run() over enterWith(). enterWith() has no boundary, so it leaks back into the caller.
There is a small cost. Activating AsyncLocalStorage turns on promise hooks for the whole process, so every promise gets a little slower. For a request that already spends milliseconds in I/O that is noise; in a hot loop, read the store once at the top and pass the value down from there.

Wrapping up

If you are threading a request id, a tenant id, a trace id or a logged-in user through function signatures that don't care about any of it, AsyncLocalStorage removes that parameter entirely. Your service and repository functions go back to taking only the arguments they actually use, and your logs stay correlated anyway.

It is built into Node, needs no dependency, and in a real app the whole integration is usually one middleware.

I have put a full working example with Express in this repo — the same API written both ways, plus runnable demos for each of the gotchas above:

https://github.com/tarun-koshti/Node-Async-Local-Storage-Example

If you have used AsyncLocalStorage in production, I would love to hear where it helped you, or where it bit you. Drop a comment.

Thanks for reading!