DEV Community: Tarun Koshti

Beyond Caching: Cron Jobs with Redis, Node.js and BullMQ

Tarun Koshti — Sun, 18 Oct 2026 10:17:09 +0000

Hello Engineers, in this article we will see how we can use Redis as a cron service to trigger scripts or APIs at a specific time.
In my last article, I showed how we can use Redis as a job scheduler to schedule delayed jobs and retry failed operations; you can check it here.

To be precise, Redis itself doesn't have a cron feature. A job library like BullMQ (for Node.js) builds it on top of Redis: it stores the schedule in Redis and always keeps the next run waiting there as a delayed job.
We'll build the example with Node.js (TypeScript) and BullMQ: a report generator that runs every night at midnight, aggregates the day's orders, and saves the result in MongoDB.

Part 1: Why Redis?

1. Single platform

If you are already using Redis in your application stack, you can use it as a cron service as well; no additional service is required.

2. Cost

Let's say you are a small enterprise, your workload is simple, and you are already using Redis. In that case, you can extend the Redis functionality and use it as a cron service.
You don't need to pay extra for any third-party service.

3. Runs once, even with many app instances

A cron library like node-cron runs inside your app process. If you run 3 instances of your app, the job runs 3 times: 3 reports, 3 emails. With BullMQ, the schedule lives in Redis and each run is given to exactly one worker, so it runs once, however many instances you have.

4. Survives restarts and deploys

An in-process schedule (node-cron, setInterval) lives in memory. If your app is down at the scheduled time, that run never happens. A BullMQ schedule is stored in Redis, so a deploy or a crash doesn't lose it, and the pending run starts as soon as a worker is back. (For this, Redis itself must keep the data across a restart; we'll configure that in Part 2.)

5. Retries built in

If a run fails, for example because the database is down for a moment, BullMQ retries it with attempts and backoff, the same as any other job. A plain cron just runs your script; if it fails, it stays failed unless you build the retry yourself.

6. You can see what is going on

The schedule and its next run time are stored in Redis, so you can check them any time with queue.getJobSchedulers(). Every run is a normal job, so completed and failed runs are kept as well, and you can see what ran and why something failed.

7. Time zone support

You can give the schedule a time zone: { pattern: '0 0 * * *', tz: 'Asia/Kolkata' } runs at midnight in India, whatever time zone your server uses. A server's crontab uses the server's time zone by default, which is often UTC in the cloud.

One thing to know before you start: if your app is down when a run is due, BullMQ runs it once when the app comes back, but it doesn't run every missed one. If your app was down for 3 runs, you get 1 late run, not 3. We'll see this in action later in this article.

Part 2: Setup

Let's set up everything end to end, so you can run every example yourself. If you already have Redis running, still read step 5: a cron needs two Redis settings that a default or a cache Redis may not have.

Tested with Node.js 22, BullMQ 6.3, ioredis 6.0, TypeScript 7.0 and Redis 7.4.

1. Install Node.js

Install Node.js 22 or later (a current LTS release) from nodejs.org, then check it:

node --version

2. Create the project and install the packages

mkdir bullmq-cron-demo
cd bullmq-cron-demo
npm init -y
npm install bullmq ioredis
npm install -D typescript tsx @types/node

bullmq: the job queue library. Its Job Schedulers are the cron part, and it already includes the cron parser, so you don't need node-cron or any other cron library.
ioredis: the Redis client BullMQ uses. Since BullMQ 6 it is an optional dependency, so you must install it yourself. Without it, the worker crashes on start with BullMQ could not load the optional 'ioredis' package.
typescript, @types/node: the TypeScript compiler and Node.js types.
tsx: runs .ts files directly, without a build step.

3. Add tsconfig.json

Create tsconfig.json in the project root:

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "nodenext",
    "moduleResolution": "nodenext",
    "types": ["node"],
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "noEmit": true
  },
  "include": ["src"]
}

"types": ["node"] is required since TypeScript 6, which no longer loads @types/node automatically. Without it you get errors like Cannot find name 'process'.
"noEmit": true, because tsx runs the code; tsc is only used to type-check:

npx tsc --noEmit

4. Start Redis

Pick one of the two options below. Both turn on persistence (AOF), set a memory limit with the noeviction policy, and keep the data in a Docker volume; step 5 explains why.

Option A: plain Docker command

# start Redis in the background on port 6379, with AOF on and a memory limit
docker run -d --name cron-redis -p 6379:6379 -v cron-redis-data:/data redis:7-alpine redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy noeviction

# check it is up (should print PONG)
docker exec cron-redis redis-cli ping

# when you are done: stop and remove the container (the data stays in the volume)
docker stop cron-redis
docker rm cron-redis

Option B: Docker Compose

Create docker-compose.yaml in the project root. The mongo service is only needed for the full order report app later in this article; the Redis-only examples work without it.

services:
  redis:
    image: redis:7-alpine
    # Turn on the append-only file, so scheduled jobs survive a restart or crash
    # (the image's default is periodic snapshots only). Cap memory at 256 MB and
    # never evict keys when it's full, so jobs and schedules are never deleted.
    command: redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy noeviction
    ports:
      - "6379:6379"
    volumes:
      - redis-data:/data

  mongo:
    image: mongodb/mongodb-community-server:8.0-ubi9
    ports:
      - "27017:27017"
    volumes:
      - mongo-data:/data/db

volumes:
  mongo-data:
  redis-data:

Then:

# start Redis (and MongoDB) in the background
docker compose up -d

# check Redis is up (should print PONG)
docker compose exec redis redis-cli ping

# when you are done: stop it (add -v to also delete the stored data)
docker compose down

Port 6379 already in use? Another Redis is probably running on your machine. Stop it, or map a different host port (-p 6380:6379, or "6380:6379" in the compose file) and use that port in your connection settings.

5. Configure Redis for a cron

The schedule, its next run, and the history of past runs are just keys in Redis. Two settings decide whether those keys can disappear: persistence (what happens on a restart) and the eviction policy (what happens when memory is full).

5.1 Persistence: turn on AOF

Redis keeps its data in memory. By default it saves it to disk only now and then, as snapshots: after 1 hour if at least 1 key changed, after 5 minutes if 100 changed, after 1 minute if 10,000 changed. The official Docker image uses the same default, and the append-only file (AOF) is off. So if Redis restarts, it can come back without your schedule.

Your app won't notice. It creates the schedule only when it starts, so until the next app restart, no run happens and no error is shown.

With AOF on, Redis writes every change to a file on disk and replays it after a restart. In Docker, that is the --appendonly yes in the commands above. In your own redis.conf:

appendonly yes
appendfsync everysec   # write to disk every second: a crash loses at most ~1 second

everysec is already the default; it's there so you know the trade-off.

5.2 Memory limit and eviction policy

What is eviction? When Redis reaches its memory limit, it can free memory by deleting keys. That is called eviction. Two settings control it:

maxmemory is the memory limit. The default is 0, which means no limit: Redis keeps using memory until the container or machine runs out.
maxmemory-policy decides what Redis does when it reaches that limit.

Redis 7 has 8 policies:

Policy	When memory is full, Redis...	Safe for a cron?
`noeviction`	deletes nothing, and rejects new writes with an error	Yes
`allkeys-lru`	deletes the least recently used keys	No
`allkeys-lfu`	deletes the least frequently used keys	No
`allkeys-random`	deletes random keys	No
`volatile-lru`	deletes the least recently used keys that have a TTL	No
`volatile-lfu`	deletes the least frequently used keys that have a TTL	No
`volatile-random`	deletes random keys that have a TTL	No
`volatile-ttl`	deletes the keys with a TTL that expire soonest	No

Why noeviction? A nightly schedule is used once a day, so for an allkeys-* policy it looks like the oldest, least used key in Redis: the first one to go. I tested it: with allkeys-lru and a 3 MB limit, I filled Redis with cache keys. Redis evicted 1,937 keys, and every key of the schedule was among them. queue.getJobSchedulers() returned an empty list, and there was no error anywhere. The cron simply stopped.

The volatile-* policies look safer, because the schedule has no TTL. But BullMQ also uses some keys with a TTL, for example the lock on a running job. If Redis evicts that lock, BullMQ can treat the job as stalled and run it again. So use noeviction for a cron, nothing else.

What happens with noeviction when memory is full? Redis rejects new writes with OOM command not allowed when used memory > 'maxmemory'. I tested this too: queue.add() failed with that error and the worker logged it, but nothing was deleted. The schedule was still there, and the runs went back to normal as soon as memory was freed. An error you can see is much better than a cron that silently disappears.

Some things to keep in mind:

Set maxmemory (the --maxmemory 256mb above). Without a limit, Redis grows until the container or machine runs out of memory, and the process can be killed instead of giving you the clear OOM error. Pick a size that fits your data with headroom; 256 MB is just for this demo.
Clean up finished runs. By default, BullMQ keeps completed and failed jobs forever, and every run of a cron is a new job. Set removeOnComplete and removeOnFail on the job (we'll do this in the example), so old runs don't fill your memory.
A separate database is not enough. If you share one Redis instance between your cache and your cron, putting the cron in another database (db: 1) doesn't help: the eviction policy applies to the whole instance. Use a separate Redis instance, or set the shared one to noeviction.
TTLs still work. noeviction only stops eviction. Keys with a TTL still expire as normal, because expiry is not eviction.

How to set it. In Docker, add the flags to the command, as in both options above: --maxmemory 256mb --maxmemory-policy noeviction. In your own redis.conf:

maxmemory 256mb
maxmemory-policy noeviction

You can also change it on a running Redis with redis-cli CONFIG SET maxmemory-policy noeviction, but that change is lost on the next restart. Put it in the command or in redis.conf as well.

6. Check your Redis

Run these against your Redis (with Docker Compose, prefix them with docker compose exec redis):

redis-cli CONFIG GET appendonly         # should be: yes
redis-cli CONFIG GET maxmemory-policy   # should be: noeviction
redis-cli CONFIG GET maxmemory          # 0 = no limit; set one in production
redis-cli INFO memory                   # used_memory_human = how much is used now
redis-cli INFO stats                    # evicted_keys should stay 0

7. Using a managed Redis?

Managed services (AWS ElastiCache, Google Memorystore, Azure Cache for Redis, and so on) handle persistence in their own way, and their default eviction policy is not always noeviction. Check both settings in your provider's configuration before you put a cron on it.

Part 3: Code Example

Now let's see the cron in action. We'll write three small files: a scheduler that creates the cron, a worker that runs it, and a shared file with the names. Redis only, no database yet.

So you don't have to wait until midnight, this example runs every 10 seconds. The real order report in the next part runs every night; only the pattern is different.

Project structure

By the end of this part, your project will look like this:

bullmq-cron-demo/
  docker-compose.yaml     (only if you used Option B in Part 2)
  package.json
  tsconfig.json
  src/
    connection.ts
    cronQueue.ts
    cronScheduler.ts
    cronWorker.ts

1. The shared files

connection.ts holds the Redis connection:

// src/connection.ts
export const connection = { host: '127.0.0.1', port: 6379 };

cronQueue.ts holds the names that the scheduler and the worker must agree on:

// src/cronQueue.ts
// Shared names for the cron example (cronScheduler.ts + cronWorker.ts). Uses
// the same Redis connection as the other examples (connection.ts).
export const CRON_QUEUE_NAME = 'demo-reports';

// The scheduler's id. Upserting the same id again updates this one scheduler;
// a different id creates a second one.
export const SCHEDULER_ID = 'demo-report';

2. The scheduler

In BullMQ, a cron is called a Job Scheduler. You create it with queue.upsertJobScheduler(). The scheduler is stored in Redis, and BullMQ always keeps its next run waiting there as a normal delayed job. When that run starts, BullMQ adds the one after it.

// src/cronScheduler.ts
import { Queue } from 'bullmq';
import { connection } from './connection';
import { CRON_QUEUE_NAME, SCHEDULER_ID } from './cronQueue';

const queue = new Queue(CRON_QUEUE_NAME, { connection });
queue.on('error', (err) => console.error('queue error:', err.message));

// Creates, lists or removes a recurring job (a Job Scheduler).
//   npx tsx src/cronScheduler.ts         -> create or update it
//   npx tsx src/cronScheduler.ts list    -> show all schedulers
//   npx tsx src/cronScheduler.ts remove  -> stop it for good
async function main() {
  const [action] = process.argv.slice(2);

  if (action === 'remove') {
    // Removes the scheduler and its next pending run. Nothing runs after this.
    await queue.removeJobScheduler(SCHEDULER_ID);
    console.log(`removed scheduler ${SCHEDULER_ID}`);
  } else if (action === 'list') {
    const schedulers = await queue.getJobSchedulers();
    for (const s of schedulers) {
      console.log(`${s.key}: '${s.pattern}' (${s.tz}), next run ${new Date(s.next!).toLocaleTimeString()}`);
    }
  } else {
    // upsertJobScheduler() = "insert or update". Run this script as many times
    // as you like: there is still only ONE scheduler with this id in Redis.
    //   arg 1 — scheduler id: the key that makes the upsert safe to repeat.
    //   arg 2 — when to run:
    //           `pattern` is a cron expression. This one has 6 fields, the first
    //           is seconds: '*/10 * * * * *' = every 10 seconds. With 5 fields
    //           ('0 0 * * *' = every day at midnight) there are no seconds.
    //           `tz` is the time zone the pattern is read in.
    //   arg 3 — the job template: every run is a normal job with this name,
    //           data and options (here: BullMQ's built-in retry).
    const firstJob = await queue.upsertJobScheduler(
      SCHEDULER_ID,
      { pattern: '*/10 * * * * *', tz: 'Asia/Kolkata' },
      {
        name: 'report',
        data: { type: 'order-summary' },
        opts: { attempts: 3, backoff: { type: 'exponential', delay: 1000 } },
      },
    );
    console.log(`scheduler ${SCHEDULER_ID} saved, next run: ${firstJob.id}`);
  }

  // The schedule lives in Redis, not in this process, so the script can exit.
  await queue.close();
}

main();

A few things to notice:

The script exits. Unlike node-cron, nothing has to keep running for the schedule to exist. It is stored in Redis, and any worker connected to that Redis will run it.
It's an upsert, not an add. The scheduler id (demo-report) is the key. Running the script again updates the same scheduler instead of creating a second one. That's why it's safe to call this on every app start, as we'll do in the real app.
Every run is a normal job. So everything you know about jobs works here too: retries with attempts + backoff, removeOnComplete / removeOnFail, and the failed event.

3. The worker

A scheduled run is a normal job, so a normal Worker processes it. Nothing cron-specific here, except one field:

// src/cronWorker.ts
import { Worker } from 'bullmq';
import { connection } from './connection';
import { CRON_QUEUE_NAME } from './cronQueue';

// A scheduled run is a normal job, so a normal Worker processes it. Start this
// in one or more terminals: each run is still processed only once, by one of
// them.
const worker = new Worker(
  CRON_QUEUE_NAME,

  async (job) => {
    // `opts.prevMillis` is the time this run was SCHEDULED for. Use it (not
    // "now") to decide what period the run covers: a late run or a retry then
    // still covers the right period.
    const scheduledFor = new Date(job.opts.prevMillis ?? job.timestamp);
    console.log(
      `[${new Date().toLocaleTimeString()}] ${job.name} for ${scheduledFor.toLocaleTimeString()}:`,
      job.data,
    );
  },

  { connection },
);

worker.on('failed', (job, err) => console.error(`  failed: ${job?.id} — ${err.message}`));
worker.on('error', (err) => console.error('  worker error:', err.message));

job.opts.prevMillis is the time the run was scheduled for, not the time it actually started. They are usually the same, but not always: if your worker was busy, restarting, or the run is being retried, it starts later. For a report, "which day does this run cover?" must come from the scheduled time, otherwise a run that starts at 00:00:03 or a retry at 00:05 could report the wrong period.

4. Run it

Start the worker in one terminal and keep it running:

npx tsx src/cronWorker.ts

In a second terminal, create the scheduler:

npx tsx src/cronScheduler.ts

scheduler demo-report saved, next run: repeat:demo-report:1792310610000

The first run's job id is repeat:<scheduler id>:<run time in milliseconds>. Because the run time is part of the id, the same run can never be added twice.

Now the worker prints a run every 10 seconds:

[1:33:30 PM] report for 1:33:30 PM: { type: 'order-summary' }
[1:33:40 PM] report for 1:33:40 PM: { type: 'order-summary' }
[1:33:50 PM] report for 1:33:50 PM: { type: 'order-summary' }

5. Run the scheduler again

Run npx tsx src/cronScheduler.ts a second time, and then list the schedulers:

npx tsx src/cronScheduler.ts
npx tsx src/cronScheduler.ts list

scheduler demo-report saved, next run: repeat:demo-report:1792310610000
demo-report: '*/10 * * * * *' (Asia/Kolkata), next run 1:33:30 PM

Same next run, and still one scheduler. The worker keeps printing one line every 10 seconds, not two.

6. Start a second worker

Start npx tsx src/cronWorker.ts in a third terminal. This is what happens when you run several instances of your app. Here's what both workers printed:

worker 1: [1:34:00 PM] report for 1:34:00 PM: { type: 'order-summary' }
worker 2: [1:34:10 PM] report for 1:34:10 PM: { type: 'order-summary' }
worker 1: [1:34:20 PM] report for 1:34:20 PM: { type: 'order-summary' }
worker 2: [1:34:30 PM] report for 1:34:30 PM: { type: 'order-summary' }

Each run went to exactly one of them. With node-cron in two app instances, every line would show up twice.

7. Stop it

npx tsx src/cronScheduler.ts remove
npx tsx src/cronScheduler.ts list

removed scheduler demo-report

The list is empty and the workers stop printing. Stopping the workers alone would not do this: the scheduler stays in Redis, and its next run waits there until a worker comes back.

Cron pattern cheat sheet

A pattern has 5 fields: minute, hour, day of month, month, day of week. Add a sixth field at the front for seconds. The times below are in the scheduler's tz.

Pattern	Runs
`/10 * * * *`	every 10 seconds (6 fields)
`/15 * * *`	every 15 minutes
`0 * * * *`	every hour, on the hour
`0 0 * * *`	every day at midnight
`30 9 * * 1-5`	Monday to Friday at 09:30
`0 0 * * 0`	every Sunday at midnight
`0 0 1 * *`	the first day of every month

If you only need a fixed interval and don't care about clock times, use every (in milliseconds) instead of pattern, for example { every: 60_000 }. You can't use both in the same scheduler.

Part 4: Real example: a daily order report

Now let's see how this looks in a real app. It's the Express + MongoDB order app from my last article: customers create orders, pay for them, and BullMQ reminds them while an order is unpaid. Now we add a cron to it, with this rule:

Every night at midnight (India time), count the orders of the last 24 hours by status, sum their amounts, count the reminders that failed, and save the result as one document in the analytics collection.

The full source code is on GitHub (link at the end). Here are the key parts.

1. The schedule config

Like the queue settings in the last article, the schedule lives in one config file, so you can change it without touching the logic:

// src/constants/appConstants.ts
export const SCHEDULER_CONFIG: Record<SchedulerName, SchedulerSettings> = {
  [SCHEDULER_NAMES.ORDER_REPORTS]: {
    schedulerId: 'daily-order-report',
    jobName: 'order-report',
    // Cron fields: minute hour day-of-month month day-of-week.
    // '0 0 * * *' = every day at 00:00 in `tz`. For testing, use '* * * * *'
    // (every minute) and set periodMs to 60_000.
    pattern: '0 0 * * *',
    tz: 'Asia/Kolkata',
    // One report covers the `periodMs` before its scheduled run time. Keep it
    // at 7 days or less: failed reminders are only kept for 7 days.
    periodMs: 24 * 60 * 60 * 1000,
    attempts: 3,
    backoff: { type: 'exponential', delay: 1_000 },
    removeOnComplete: { age: 7 * 24 * 60 * 60 },
    removeOnFail: { age: 30 * 24 * 60 * 60 },
  },
};

pattern + tz: every day at 00:00, India time, whatever time zone the server uses.
periodMs: how much time one report covers. It must match the pattern: a daily pattern covers 24 hours. If you change one, change the other.
attempts + backoff: the same built-in retry as any other job.
removeOnComplete / removeOnFail: every run is a new job, so without these, BullMQ would keep every run in Redis forever. Here completed runs are kept for 7 days and failed ones for 30 days, so you have time to debug them.

2. Create the schedule on every app start

The app creates the schedule when it starts. Because it's an upsert, this is safe on every restart, and in every instance:

// src/services/queue.ts
export async function upsertScheduler(name: SchedulerName) {
  const { schedulerId, jobName, pattern, tz, attempts, backoff, removeOnComplete, removeOnFail } =
    SCHEDULER_CONFIG[name];
  return getQueue(name).upsertJobScheduler(
    schedulerId,
    { pattern, tz },
    { name: jobName, opts: { attempts, backoff, removeOnComplete, removeOnFail } },
  );
}

// src/index.ts (main)
// Safe on every start: upserting keeps one schedule, however many times (or
// in however many instances) the app starts.
await upsertScheduler(SCHEDULER_NAMES.ORDER_REPORTS);

With this, the schedule always follows the code: change the pattern in the config, restart the app, and the existing scheduler is updated.

3. The worker

A normal worker on the report queue, running in the same process as the API (in production you could run it as its own process):

// src/services/reportWorker.ts
export const reportWorker = new Worker(SCHEDULER_NAMES.ORDER_REPORTS, processOrderReport, {
  connection: config.redis,
});

4. The processor: the right period, saved once

// src/services/orderReport.ts
export async function processOrderReport(job: Job): Promise<void> {
  // The period ends at the run's SCHEDULED time, not at "now": a run that
  // starts late, or a retry, still reports the same period. A Job Scheduler
  // stores that time in `opts.prevMillis`; a manual run (added with
  // queue.add) uses the time it was added.
  const periodEnd = new Date(job.opts.prevMillis ?? job.timestamp);
  const periodStart = new Date(periodEnd.getTime() - periodMs);

  const [orders, failedReminders] = await Promise.all([
    summariseOrders(periodStart, periodEnd),
    countFailedReminders(periodStart, periodEnd),
  ]);

  // Upsert on (reportType, periodEnd): running the same period twice
  // overwrites the report instead of creating a second one.
  const report = await AnalyticsModel.findOneAndUpdate(
    { reportType: REPORT_TYPE, periodEnd },
    {
      periodStart,
      tz,
      ...orders,
      failedReminders,
      trigger: job.repeatJobKey ? 'schedule' : 'manual',
      generatedAt: new Date(),
    },
    { upsert: true, new: true },
  );
  // ... logs the report
}

Two things are happening here:

The period comes from the scheduled time. The midnight run reports 00:00 yesterday to 00:00 today, even if it starts a few seconds late or is retried 5 minutes later. If it used new Date() instead, a retry would report a slightly different period, and some orders would be counted twice or not at all.
The save is an upsert. If the same period is processed again (a retry after the save, or a manual re-run), the report is overwritten, not duplicated. It's the same idea as the deterministic job id in the last article: make doing it twice harmless.

job.repeatJobKey is only set on runs created by a scheduler, so the report also records whether it came from the schedule or from a manual run.

5. What goes into the report

The orders are counted with one MongoDB aggregation:

// src/services/orderReport.ts
// Orders created in the period, counted and summed by their current status.
async function summariseOrders(from: Date, to: Date) {
  const groups: { _id: 'pending' | 'paid'; count: number; amount: number }[] =
    await OrderModel.aggregate([
      { $match: { createdAt: { $gte: from, $lt: to } } },
      { $group: { _id: '$status', count: { $sum: 1 }, amount: { $sum: '$amount' } } },
    ]);

  const orders = { total: 0, pending: 0, paid: 0 };
  const amount = { total: 0, pending: 0, paid: 0 };
  for (const group of groups) {
    orders[group._id] = group.count;
    amount[group._id] = group.amount;
    orders.total += group.count;
    amount.total += group.amount;
  }
  return { orders, amount };
}

The failed reminders come from BullMQ itself. A reminder job that failed all its tries stays in the queue's failed set, with the time it failed in finishedOn:

// src/services/orderReport.ts
// Reminder jobs that failed all their tries during the period. BullMQ keeps
// them in the queue's failed set (for 7 days, see `removeOnFail`), with the
// time they failed in `finishedOn`.
async function countFailedReminders(from: Date, to: Date): Promise<number> {
  const failed = await getFailedJobs(QUEUE_NAMES.ORDER_REMINDERS);
  return failed.filter(
    (job) => job.finishedOn && job.finishedOn >= from.getTime() && job.finishedOn < to.getTime(),
  ).length;
}

That's why the config says to keep periodMs at 7 days or less: the reminder queue keeps failed jobs for only 7 days, so a longer period would miss some of them.

6. One report per period

A unique index backs up the upsert from step 4: even if two runs of the same period save at the same moment, MongoDB keeps only one document for that period:

// src/models/analytics.ts
// One report per type and period. The job saves with an upsert on this key, so
// a retry or a second run overwrites the report instead of adding a duplicate.
analyticsSchema.index({ reportType: 1, periodEnd: 1 }, { unique: true });

7. The API

Three small endpoints, so you can see what the cron is doing (the code is in the repo):

Endpoint	What it does
`GET /reports`	The latest reports, newest first (`?limit=`, default 10)
`GET /reports/schedule`	The schedule as stored in Redis, with its next run time
`POST /reports/run`	Runs a report now, outside the schedule; returns `202` + job id

Run it

Clone the repo and start it (the README has the details):

docker compose up -d
cp .env.example .env
npm install
npm run dev

So you don't wait until midnight, set the testing values first, as the comment in appConstants.ts says: pattern: '* * * * *' and periodMs: 60_000. Then create three orders and pay for one:

curl -X POST localhost:3000/orders -H "Content-Type: application/json" -d '{"item":"book","amount":100}'
curl -X POST localhost:3000/orders -H "Content-Type: application/json" -d '{"item":"pen","amount":20}'
curl -X POST localhost:3000/orders -H "Content-Type: application/json" -d '{"item":"bag","amount":500}'
curl -X POST localhost:3000/orders/<id of the bag order>/pay

At the next full minute, the app logs the report (the reminder lines come from the reminder feature of the last article):

MongoDB connected
Redis connected
Order report scheduled: '* * * * *' (Asia/Kolkata), next run 2026-10-18T08:21:00.000Z
API listening on http://localhost:3000
Reminder delay: 30000ms
Reminder 1/3: order 6ad48147f3aa32fe4a0013d4 is still unpaid
Reminder 1/3: order 6ad48147f3aa32fe4a0013d7 is still unpaid
Order report 2026-10-18T08:20:00.000Z to 2026-10-18T08:21:00.000Z: {"orders":{"total":3,"pending":2,"paid":1},"amount":{"total":620,"pending":120,"paid":500},"failedReminders":0}

And it's saved in MongoDB:

curl "localhost:3000/reports?limit=1"

{
  "reports": [
    {
      "orders": { "total": 3, "pending": 2, "paid": 1 },
      "amount": { "total": 620, "pending": 120, "paid": 500 },
      "_id": "6ad4816caf408fe868b6b431",
      "reportType": "order-summary",
      "periodEnd": "2026-10-18T08:21:00.000Z",
      "__v": 0,
      "failedReminders": 0,
      "generatedAt": "2026-10-18T08:21:00.069Z",
      "periodStart": "2026-10-18T08:20:00.000Z",
      "trigger": "schedule",
      "tz": "Asia/Kolkata"
    }
  ]
}

Notice the period: exactly 08:20:00 to 08:21:00, the scheduled minute, even though the report was generated at 08:21:00.069.

A manual run goes through the same queue and worker:

curl -X POST localhost:3000/reports/run

{"jobId":"21"}
Order report 2026-10-18T08:20:00.134Z to 2026-10-18T08:21:00.134Z: {"orders":{"total":3,"pending":2,"paid":1},"amount":{"total":620,"pending":120,"paid":500},"failedReminders":0}

A manual run has no scheduled time, so its period ends when it was added (08:21:00.134). When you are done testing, set the pattern and periodMs back to the daily values.

Part 5: Gotchas

Everything in this part I hit or tested while building the example. None of these shows an error, which is exactly why they are worth knowing before you go to production.

First, one thing about how BullMQ works, because most of the gotchas come from it: a scheduler does not run anything by itself. It always keeps exactly one next run waiting in Redis. When a worker picks up that run, the worker adds the next one. So the schedule only moves forward while a worker is running.

1. Missed runs are not caught up

I stopped the app at 10:56:16 and started it again at 10:59:07, with the every-minute testing pattern. The runs for 10:57, 10:58 and 10:59 were missed. This is what the app logged on start:

Order report 2026-10-10T10:56:00.000Z to 2026-10-10T10:57:00.000Z: {"orders":{"total":0,"pending":0,"paid":0},...}
Order report scheduled: '* * * * *' (Asia/Kolkata), next run 2026-10-10T11:00:00.000Z

One late run (the one that was already waiting in Redis), then straight to 11:00. The other missed runs never happen, so those periods have no report. It doesn't matter how long the app was down: a week later, the app started with one 8-day-old run, and then continued normally.

For many crons this is what you want: a cleanup job doesn't need to run 7 times after a week of downtime. But for a report, a gap is a missing day.

What to do: if every period must have a result, don't rely on the cron alone. For example, on start (or in the run itself), compare the latest saved report with the periods that should exist, and generate the missing ones. Because our report is saved with an upsert on its period, generating a period again is harmless.

2. A slow run can overlap the next one

What if a run takes longer than the time between two runs? I tested it with a scheduler every 5 seconds, a run that takes 12 seconds, and two workers:

2:40:05 PM worker 1 START run for 2:40:05 PM
2:40:10 PM worker 2 START run for 2:40:10 PM
2:40:17 PM worker 1 END   run for 2:40:05 PM

From 2:40:10 to 2:40:17, two runs were running at the same time. With several app instances (or a worker with concurrency above 1), this is normal: each run is a separate job, and a free worker takes it.

What to do: either make the run safe to overlap (like our upsert), or allow only one job at a time on that queue:

await queue.setGlobalConcurrency(1);

Same test with that line:

2:41:05 PM worker 2 START run for 2:41:05 PM
2:41:17 PM worker 2 END   run for 2:41:05 PM
2:41:17 PM worker 2 START run for 2:41:10 PM
2:41:29 PM worker 2 END   run for 2:41:10 PM
2:41:29 PM worker 2 START run for 2:41:20 PM

No overlap any more, but look at the times: the runs start late, and the run for 2:41:15 never happened. That's gotcha 1 again: when the run for 2:41:10 was finally picked up at 2:41:17, BullMQ added the next run for the next matching time after that moment, which was 2:41:20. The global concurrency applies to the whole queue, so give the cron its own queue, as we did.

3. Changing the scheduler id leaves the old one running

Changing the pattern is safe: the upsert updates the existing scheduler. But if you change the id (for example daily-order-report → daily-order-report-v2), BullMQ sees a new scheduler, and the old one stays in Redis. This is what queue.getJobSchedulers() returned after I upserted a new id:

daily-order-report-v2 '0 0 * * *'
daily-order-report '*/5 * * * *'

Two schedulers on the same queue, and both keep running. With the same pattern, every report would run twice.

What to do: treat the scheduler id like a database key, and don't rename it casually. If you have to, remove the old one:

await queue.removeJobScheduler('daily-order-report');

4. Deleting the code doesn't delete the schedule

The schedule lives in Redis, not in your code. If you remove the upsertJobScheduler() call from your app, or stop all workers, the scheduler is still there. The next time any worker connects to that queue, it starts running again. That's how the 8-day-old run in gotcha 1 happened.

What to do: to really stop a cron, call queue.removeJobScheduler(id), as in the remove command of Part 3.

5. Change the pattern and the period together

Our report covers the periodMs before its scheduled time. When I changed the pattern from every minute to every 5 minutes but left periodMs at 60 seconds, the 11:00 run reported this:

Order report 2026-10-10T10:59:00.000Z to 2026-10-10T11:00:00.000Z: ...

Only the last minute of the 5. With these settings, 4 of every 5 minutes would never be in any report.

What to do: keep the two values next to each other in your config with a comment, like our appConstants.ts, and change them together.

6. Daylight saving time

India has no daylight saving time, so Asia/Kolkata is easy. In a time zone that has it, the clock jumps. I checked a daily schedule in America/New_York with the same cron parser BullMQ uses (in 2026, the clocks jump forward on 8 March and back on 1 November):

'30 2 * * *' (02:30):
   Sat, Mar 07, 02:30 AM EST
   Sun, Mar 08, 03:30 AM EDT    <- 02:30 doesn't exist that day, so it runs at 03:30
   Mon, Mar 09, 02:30 AM EDT

'0 0 * * *' (midnight):
   Sun, Mar 08, 12:00 AM EST
   Mon, Mar 09, 12:00 AM EDT    <- only 23 hours after the previous run

And when the clocks go back, a 01:30 run (a time that happens twice that night) runs only once.

So the runs themselves are fine, but on those two days a "day" is 23 or 25 hours. A fixed periodMs of 24 hours would then miss one hour, or count one hour twice.

What to do: if your time zone has daylight saving time, don't schedule in the hours when the clocks change (in the US, between 01:00 and 03:00), and compute the period from the previous scheduled run instead of a fixed periodMs.

7. Redis settings

The last gotcha is the one we fixed in Part 2: with the wrong eviction policy, Redis deletes the schedule when memory fills up, and without AOF, a Redis restart can lose it. In both cases, the cron just stops, with no error. Check appendonly and maxmemory-policy on every Redis you put a cron on.

Wrapping up

In this article, we saw:

why Redis (with BullMQ) is a good fit for a cron (Part 1),
how to set up Node.js and Redis end to end, including persistence and the eviction policy (Part 2),
how to create, run, list and remove a Job Scheduler (Part 3),
how this looks in a real Express app: a daily order report (Part 4),
and the gotchas that don't show any error (Part 5).

If you already run Redis, you probably don't need a separate cron service: one upsertJobScheduler() call gives you a schedule that runs once across all your app instances, survives restarts, and retries on failure.

When is it not the right tool? If you don't run Redis at all, adding it only for a few cron jobs may be more work than a managed scheduler from your cloud provider. And if every single run must happen, even after downtime, remember gotcha 1: you'll need to fill the gaps yourself.

The complete source code is on GitHub: tarun-koshti/Redis-BullMq

Thanks for reading!

Beyond Caching: Scheduling Jobs with Redis, Node.js and BullMQ

Tarun Koshti — Fri, 09 Oct 2026 10:21:31 +0000

Hello Engineers, in this article we will see how Redis can be used as a job scheduler.
Most of us mainly use Redis to cache very frequently accessed data, but beyond data caching, Redis has lots of other applications as well. Some of them are listed below:

As a scheduler (topic of the current article)
As a cron service
As a message queue like SQS
For message delivery with publish/subscribe (Pub/Sub) in real-time applications

So in this article, we will see its job scheduler use case.
We'll build the examples with Node.js (TypeScript) and BullMQ, but the concepts apply to any language with a Redis job library.

Part 1: Why Redis?

1. Cost perspective

Say you're a small or medium-sized business whose application handles up to a million requests a day, and you already use Redis for caching. Adding a job queue costs you almost nothing: no new service to provision, no new bill, and no new system for your team to learn and monitor. At roughly 12 requests per second on average, your workload uses a tiny fraction of what a single Redis instance can handle. (Reusing your cache instance? See the configuration tip below.)

2. Precise and flexible timing

Jobs fire within milliseconds of their scheduled time, and delays can range from a few milliseconds to weeks. Many managed alternatives have limits here. AWS SQS, for example, caps message delays at 15 minutes.

3. Speed

Because Redis works in memory, scheduling and claiming a job typically takes under a millisecond. A single instance can handle tens of thousands of job operations per second, which is far more than most applications need.

4. Mature ecosystem

You rarely need to build a scheduler yourself. Battle-tested libraries exist for most languages: BullMQ for Node.js, Sidekiq for Ruby, Celery and RQ for Python, and Asynq for Go. They provide retries with backoff, attempt limits, priorities, rate limiting, repeatable jobs, and monitoring dashboards out of the box.

5. Simple mental model

The core pattern is easy to understand and debug: a sorted set where each job's score is derived from the time it should run. You can inspect pending jobs with a single command (ZRANGE bull:<queue>:delayed 0 -1 WITHSCORES), which makes troubleshooting straightforward compared to black-box managed services. (BullMQ stores the run time multiplied by 4096, so divide a score by 4096 to get the timestamp in milliseconds.)

6. One tool, many jobs

Job scheduling usually doesn't come alone. You often also need rate limiting (so you don't flood a customer's webhook endpoint), distributed locks, deduplication, and caching. Redis handles all of these, so a single piece of infrastructure covers several needs.

7. Portability

Redis runs anywhere: locally in Docker, on any cloud, or through managed services like AWS ElastiCache, Google Memorystore, and Azure Cache for Redis. Unlike cloud-specific options such as Google Cloud Tasks or AWS SQS, it doesn't lock you into one provider.

Configuration tip: memory policy

If you reuse your caching Redis for jobs (see point 1), check one setting first: caches and queues want opposite settings. A cache typically uses an eviction policy like allkeys-lru, which deletes old keys when memory fills up. A queue needs noeviction, or your jobs can silently disappear. So either run the queue on a separate Redis instance, or make sure the shared instance is configured for the queue's needs. (A separate database inside the same instance is not enough: the eviction policy applies to the whole instance, so cache traffic can still evict your jobs.)

What does that setting actually mean?

maxmemory-policy is the Redis setting that decides what happens when Redis reaches its memory limit (maxmemory). Options include allkeys-lru (delete the least recently used keys), volatile-lru (delete the least recently used keys that have a TTL), and others.
noeviction means Redis never deletes data to free memory. Once the limit is reached, it rejects new writes with an error (OOM command not allowed when used memory > 'maxmemory'), but keeps everything already stored. That's the safe behavior for a job queue, since your jobs won't vanish.
noeviction is Redis's own built-in default, so a fresh Redis (including the official Docker image) is already queue-safe. The risk comes when a shared instance is switched to allkeys-lru for caching.
Two things to keep in mind: keys that have a TTL still expire as normal, because expiry is not eviction. And by default maxmemory is 0 (no limit), so Redis keeps using memory until the machine runs out. In production, set maxmemory explicitly.

Does Redis work for large applications as well? Yes, of course, but you can't run Redis as a plain instance, because at large scale durability matters: a crash or restart without the right persistence settings can lose jobs.
At this scale, a bad configuration can lose all the scheduled jobs, causing permanent data loss.
Check the "Can I use it for very large applications?" section in Part 4 for more details.

Part 2: Code Example

Now it's time to see Redis in action.
In this demo, we'll use BullMQ (a Node.js job queue library built on Redis) to schedule a job with a delay and process it later.

Workspace setup

Let's set up the workspace first, so you can run every example yourself.

Tested with Node.js 22, BullMQ 6.3, ioredis 6.0, TypeScript 7.0 and Redis 7.

1. Install Node.js

Install Node.js 22 or later (a current LTS release) from nodejs.org, then check it:

node --version

2. Create the project and install the packages

mkdir bullmq-demo
cd bullmq-demo
npm init -y
npm install bullmq ioredis
npm install -D typescript tsx @types/node

bullmq — the job queue library.
ioredis — the Redis client BullMQ uses. Since BullMQ 6 it is an optional dependency, so you must install it yourself. Without it, the worker crashes on start with BullMQ could not load the optional 'ioredis' package.
typescript, @types/node — the TypeScript compiler and Node.js types.
tsx — runs .ts files directly, without a build step.

3. Add tsconfig.json

Create tsconfig.json in the project root:

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "nodenext",
    "moduleResolution": "nodenext",
    "types": ["node"],
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "noEmit": true
  },
  "include": ["src"]
}

"types": ["node"] is required since TypeScript 6, which no longer loads @types/node automatically. Without it you get errors like Cannot find name 'process'.
"noEmit": true, because tsx runs the code; tsc is only used to type-check:

npx tsc --noEmit

4. Start Redis

BullMQ needs a running Redis server. Pick one of the two options below.

Option A — plain Docker command

# start Redis in the background on port 6379
docker run -d --name bullmq-redis -p 6379:6379 redis:7-alpine

# check it is up (should print PONG)
docker exec bullmq-redis redis-cli ping

# when you are done: stop and remove the container
docker stop bullmq-redis
docker rm bullmq-redis

Option B — Docker Compose

Create docker-compose.yaml in the project root:

services:
  redis:
    image: redis:7-alpine
    # Turn on the append-only file, so scheduled jobs survive a restart
    # (the image's default is periodic snapshots only; see Part 4)
    command: redis-server --appendonly yes
    ports:
      - '6379:6379'
    volumes:
      - redis-data:/data

volumes:
  redis-data:

Then:

# start Redis in the background
docker compose up -d

# check it is up (should print PONG)
docker compose exec redis redis-cli ping

# when you are done: stop it (add -v to also delete the stored data)
docker compose down

Port 6379 already in use? Another Redis is probably running on your machine. Stop it, or map a different host port (-p 6380:6379, or "6380:6379" in the compose file) and set port: 6380 in connection.ts.

The redis:7-alpine image uses maxmemory-policy noeviction by default, which is the setting a job queue needs (see the configuration tip in Part 1).

maxmemory-policy is the Redis setting that decides what happens when Redis reaches its memory limit (maxmemory). Options include allkeys-lru (delete the least recently used keys), volatile-lru (delete the least recently used keys that have a TTL), and others.

noeviction means Redis never deletes data to free memory. Once the limit is reached, it rejects new writes with an error (OOM command not allowed when used memory > 'maxmemory'), but keeps everything already stored.

So if you run that image without changing anything, Redis will never delete your keys to free memory. That's the safe behavior for a job queue, since your jobs won't vanish. (noeviction is actually Redis's own built-in default; the image just doesn't override it.)

Two things to keep in mind: keys that have a TTL still expire as normal, because expiry is not eviction. And by default maxmemory is 0 (no limit), so Redis keeps using memory until the container or machine runs out. In production, set maxmemory explicitly, so you get the clear OOM error above instead of the process being killed.

5. Project structure

By the end of this part, your project will look like this:

bullmq-demo/
  docker-compose.yaml     (only if you use Option B)
  package.json
  tsconfig.json
  src/
    connection.ts
    worker.ts
    scheduler.ts
    workerManualRetry.ts

The code

First, connection.ts. It contains the shared connection and the job data interface required for this demo.

// src/connection.ts
// Shared Redis connection for the article demo
export const connection = {
  host: '127.0.0.1',
  port: 6379,
};

export const QUEUE_NAME = 'demo-reminders';

// Shape of the job body (what the scheduler passes as `data`).
// The retry settings travel WITH the job, so the manual retry worker
// (workerManualRetry.ts) can decide on its own whether to reschedule and how
// long to wait.
export interface ReminderJobData {
  orderId: string;
  reason: string;
  attempt: number; // which attempt this is, 1-based
  maxAttempts: number; // how many attempts to make in total before giving up
  retryDelayMs: number; // how long to wait before the next attempt
}

Now worker.ts.
The worker.ts is basically a long-running program: it keeps listening to the queue, and processes each job once it becomes ready (here, after its delay has elapsed).

// src/worker.ts
import { Worker } from 'bullmq';
import { connection, QUEUE_NAME } from './connection';

// The WORKER is the consumer side of the queue: it pulls jobs from Redis and processes them.
// Basically the worker contains the business logic to process the job's data, like sending emails, calling APIs and the database
// It does not reschedule or mark the job itself: returning normally marks the job completed, and throwing marks it failed — BullMQ then retries it according to the job's attempts/backoff options
// Keep it running in one terminal, then schedule jobs from another.
// The worker continuously pulls jobs for processing
//
// A Worker is bound to exactly ONE queue. If you have several queues you need to create a separate worker instance for every queue name
// The first argument is the queue name (Example: "DEMO_QUEUE")
// The second argument is the async function which contains the business logic
// The third argument is the connection configuration
const worker = new Worker(
  QUEUE_NAME,
  async (job) => {
    // job.id   — unique id of this job in the queue
    // job.name — the job's name (set by queue.add('reminder', ...) in
    //            scheduler.ts); use it to route different job types handled
    //            by the same worker
    // job.data — the payload passed when the job was scheduled
    console.log('Processing queue:', QUEUE_NAME);
    console.log(`[${new Date().toLocaleTimeString()}] processing job ${job.id}:`, job.data);
  },

  { connection },
);

// Lifecycle events. The worker keeps a long-lived connection to Redis and emits
// an event for every job it finishes.
worker.on('completed', (job) => console.log(`  done: ${job.id}`)); // job succeeded
worker.on('failed', (job, err) => console.error(`  failed: ${job?.id} — ${err.message}`)); // job threw

// Redis connection problems are reported here. Without this listener, BullMQ
// prints a raw stack trace on every reconnect attempt.
worker.on('error', (err) => console.error('  worker error:', err.message));

Now scheduler.ts.
The scheduler is used to add jobs to the queue (and to cancel them before they run).
Run it to add a job to the queue.

// src/scheduler.ts
import { Queue } from 'bullmq';
import { connection, QUEUE_NAME } from './connection';

// A Queue is the producer-side handle to a queue in Redis: you use it to ADD
// jobs and to look them up / remove them. It does not process anything — that's
// the worker's job.
//   arg 1 — queue name (must match the worker's queue name)
//   arg 2 — options, including the Redis connection
const queue = new Queue(QUEUE_NAME, { connection });

// Redis connection problems are reported here. Without this listener, BullMQ
// prints a raw stack trace on every reconnect attempt.
queue.on('error', (err) => console.error('queue error:', err.message));

// The SCHEDULER enqueues a delayed job, or cancels one before it runs.
//   npx tsx src/scheduler.ts            -> schedule a job (fires in 5s)
//   npx tsx src/scheduler.ts cancel <id> -> cancel that job
async function main() {
  // Read CLI args: e.g. `... scheduler.ts cancel 42` -> action="cancel", jobId="42"
  const [action, jobId] = process.argv.slice(2);

  if (action === 'cancel') {
    // getJob(id) looks a job up by id. Returns the Job, or undefined if it has
    // already run or was removed.
    const job = await queue.getJob(jobId);
    if (job) {
      // remove() deletes the job from the queue. For a delayed job that hasn't
      // run yet, this cancels it — the worker will never see it.
      await job.remove();
      console.log(`cancelled job ${jobId} — it will not run`);
    } else {
      console.log(`no job ${jobId} found (already ran or removed)`);
    }
  } else {
    const delayMs = 5000;

    // add() puts a new job on the queue.
    //   arg 1 — job name ('reminder'): a label the worker can switch on to
    //           handle different job types on the same queue.
    //   arg 2 — data: the payload, must be JSON-serializable. The worker reads
    //           this as job.data. The `attempt` / `maxAttempts` / `retryDelayMs`
    //           fields are only used by the manual retry worker
    //           (workerManualRetry.ts); the basic worker just logs them.
    //   arg 3 — options: here `delay` (ms) holds the job back so it only becomes
    //           available to the worker after that time. Other options live here
    //           too: `attempts`, `backoff`, `jobId`, `priority`, etc. (Recurring
    //           jobs are not an add() option — use queue.upsertJobScheduler().)
    const job = await queue.add(
      'reminder',
      {
        orderId: 'order-123',
        reason: 'unpaid',
        attempt: 1,
        maxAttempts: 3,
        retryDelayMs: 3000,
      },
      { delay: delayMs },
    );
    console.log(`scheduled job ${job.id} to run in ${delayMs / 1000}s`);
    console.log(`cancel it with: npx tsx src/scheduler.ts cancel ${job.id}`);
  }

  // Close the connection so this short-lived script can exit (the worker, by
  // contrast, stays open to keep processing).
  await queue.close();
}

main();

In the above example, you can see how Redis can be used for simple job scheduling.

Now, what if your job fails during processing?
You need to retry it. In the next example, I'll show you how to retry a failed job.

Manual job retry example

Here the job body carries the retry settings, and the worker reschedules the job itself when the work fails.

// src/workerManualRetry.ts
//
// MANUAL RETRY EXAMPLE — an alternative to worker.ts.
//
// Instead of letting BullMQ retry (via `attempts` + `backoff` in the job
// options), the retry settings travel in the JOB BODY and this worker decides
// for itself whether to try again. On failure it adds a NEW job to the same
// queue with attempt + 1, delayed by retryDelayMs.
//
// Run EITHER this worker OR worker.ts — not both. They consume the same queue,
// so if both run they will compete for the same jobs.
//
// Trade-offs vs BullMQ's built-in retry:
//   * the retry is a NEW job with a NEW id (built-in retry reuses the same job
//     id and increments job.attemptsMade)
//   * because we CATCH the error, BullMQ never sees a failure — every attempt
//     is recorded as 'completed' and the 'failed' event never fires
// Use this only when you need that control; otherwise prefer `attempts` + `backoff`.
import { Queue, Worker } from 'bullmq';
import { connection, QUEUE_NAME, ReminderJobData } from './connection';

// A Queue handle, so the worker can put the NEXT attempt back on the same queue.
// Only needed because we are retrying manually.
const queue = new Queue(QUEUE_NAME, { connection });
queue.on('error', (err) => console.error('  queue error:', err.message));

// Simulated work — this is the part that can fail in real life (SMTP down,
// payment API timing out, etc.). It fails on attempt 1 and succeeds from
// attempt 2, so the retry path is easy to watch in the demo.
async function sendReminder(data: ReminderJobData): Promise<void> {
  if (data.attempt === 1) {
    throw new Error('payment provider timed out');
  }
  console.log(`  reminder sent for ${data.orderId} (${data.reason})`);
}

const worker = new Worker<ReminderJobData>(
  QUEUE_NAME,

  async (job) => {
    // The retry settings come from the job body, not the job options.
    const { attempt, maxAttempts, retryDelayMs } = job.data;

    console.log('Processing queue:', QUEUE_NAME);
    console.log(
      `[${new Date().toLocaleTimeString()}] job ${job.id} — attempt ${attempt}/${maxAttempts}`,
    );

    try {
      await sendReminder(job.data);
      console.log(`  success on attempt ${attempt}`);
    } catch (err) {
      const reason = (err as Error).message;

      if (attempt < maxAttempts) {
        // RESCHEDULE: add a NEW job to the same queue carrying an incremented
        // attempt counter, delayed by retryDelayMs. That delayed job becomes
        // "attempt 2", and this one finishes normally.
        //   arg 1 — same job name, so the same worker logic handles it
        //   arg 2 — the same body, with attempt bumped by 1
        //   arg 3 — delay: wait this long before the retry becomes available
        await queue.add(job.name, { ...job.data, attempt: attempt + 1 }, { delay: retryDelayMs });

        console.log(
          `  attempt ${attempt} failed (${reason}) — retrying in ${retryDelayMs / 1000}s as attempt ${attempt + 1}`,
        );
      } else {
        // No attempts left: stop here instead of scheduling another job.
        console.error(`  attempt ${attempt} failed (${reason}) — no attempts left, giving up`);
      }
    }
  },

  { connection },
);

// NOTE: 'completed' fires for every attempt here — including the ones whose
// work failed — because the error was caught above. 'failed' only fires for
// errors we did not catch.
worker.on('completed', (job) => console.log(`  done: ${job.id}`));
worker.on('failed', (job, err) => console.error(`  failed: ${job?.id} — ${err.message}`));

// Redis connection problems are reported here. Without these listeners, BullMQ
// prints a raw stack trace on every reconnect attempt.
worker.on('error', (err) => console.error('  worker error:', err.message));

Part 3: From demo to a real app

In Part 2, we scheduled a job, cancelled it, and retried a failed job manually.
In this part, we'll see:

The easier, recommended way to retry: BullMQ's built-in retry.
A real example from an Express app: unpaid order reminders.
A few tips before you take it to production.

Built-in retry (the recommended way)

With the manual retry, the worker caught the error and added a new job by itself.
BullMQ can do this for you: set attempts and backoff when you add the job, and simply throw from the worker when the work fails. BullMQ then retries the same job automatically.

In src/scheduler.ts, add attempts and backoff to the job options:

const job = await queue.add(
  'reminder',
  {
    orderId: 'order-123',
    reason: 'unpaid',
    attempt: 1,
    maxAttempts: 3,
    retryDelayMs: 3000,
  },
  { delay: delayMs, attempts: 3, backoff: { type: 'exponential', delay: 1000 } },
);

attempts: 3 — try the job at most 3 times in total (the first try + 2 retries).
backoff — how long to wait between tries. exponential with delay: 1000 waits 1s before the 2nd try and 2s before the 3rd (the wait doubles each time). Use type: 'fixed' to wait the same time every try.

The attempt, maxAttempts and retryDelayMs fields in the body are only used by the manual retry worker; the built-in retry ignores them.

Now the worker. Notice there is no try/catch and no re-adding of the job:

// src/workerBuiltInRetry.ts
//
// BUILT-IN RETRY EXAMPLE — the recommended way.
//
// The worker simply THROWS when the work fails. BullMQ sees the failure and
// retries the SAME job automatically, using the `attempts` and `backoff`
// options given when the job was added (see scheduler.ts).
//
// Run EITHER this worker OR worker.ts / workerManualRetry.ts — not more than
// one. They consume the same queue, so they would compete for the same jobs.
import { Worker } from 'bullmq';
import { connection, QUEUE_NAME } from './connection';

// Simulated work — fails on the first attempt and succeeds on the second, so
// the retry is easy to watch.
async function sendReminder(orderId: string, attempt: number): Promise<void> {
  if (attempt === 1) {
    throw new Error('payment provider timed out');
  }
  console.log(`  reminder sent for ${orderId}`);
}

const worker = new Worker(
  QUEUE_NAME,

  async (job) => {
    // job.attemptsMade — how many attempts have already FAILED (0 on the first run)
    // job.opts.attempts — the total attempts allowed, set when the job was added
    const attempt = job.attemptsMade + 1;
    console.log(
      `[${new Date().toLocaleTimeString()}] job ${job.id} — attempt ${attempt}/${job.opts.attempts}`,
    );

    // No try/catch here: if this throws, BullMQ marks the attempt as failed,
    // waits according to `backoff`, and runs the same job again.
    await sendReminder(job.data.orderId, attempt);
  },

  { connection },
);

// 'failed' fires for every failed attempt; 'completed' fires once, when an
// attempt finally succeeds.
worker.on('completed', (job) => console.log(`  done: ${job.id}`));
worker.on('failed', (job, err) =>
  console.error(`  failed: ${job?.id} (attempt ${job?.attemptsMade}) — ${err.message}`),
);

// Redis connection problems are reported here. Without this listener, BullMQ
// prints a raw stack trace on every reconnect attempt.
worker.on('error', (err) => console.error('  worker error:', err.message));

Run the worker in one terminal and the scheduler in another:

npx tsx src/workerBuiltInRetry.ts

npx tsx src/scheduler.ts

Output:

[8:06:41 AM] job 1 — attempt 1/3
  failed: 1 (attempt 1) — payment provider timed out
[8:06:42 AM] job 1 — attempt 2/3
  reminder sent for order-123
  done: 1

The first attempt failed, BullMQ waited 1 second, and ran job 1 again, which succeeded.

Manual retry vs built-in retry

	Manual retry (Part 2)	Built-in retry
Retry settings live in	the job body	the job options
On failure, the worker	catches the error and adds a new job	just throws
Job id on retry	a new id every time	the same id
`failed` event	never fires (every attempt is "completed")	fires for every failed attempt
Code you write	try/catch + re-add logic	nothing extra

Use the built-in retry by default. Reach for the manual retry only when you need full control over each retry, for example a different payload for every attempt.

Real example: unpaid order reminders

Now let's see how this looks in a real app. The example is an Express + MongoDB app with this rule:

When an order is created, remind the customer every 30 seconds, up to 3 times, while the order is unpaid. When the order is paid, cancel the remaining reminders.

The full source code is on GitHub (link at the end). Here are the key parts.

1. Two different "3"s in the config

// src/constants/appConstants.ts
export const QUEUE_CONFIG: Record<QueueName, QueueSettings> = {
  [QUEUE_NAMES.ORDER_REMINDERS]: {
    jobName: 'unpaid-reminder',
    delay: 30_000,
    maxNotifications: 3,
    attempts: 3,
    backoff: { type: 'exponential', delay: 1_000 },
    removeOnComplete: { age: 24 * 60 * 60 }, // seconds
    removeOnFail: { age: 7 * 24 * 60 * 60 },
  },
};

These two numbers look the same, but they mean different things:

maxNotifications is business logic: how many reminders to send while the order is unpaid.
attempts + backoff is BullMQ's built-in retry: how many times to retry one reminder if sending it fails.

The last two lines tell BullMQ when it may delete finished jobs from Redis: completed ones after 24 hours, failed ones after 7 days. By default, BullMQ keeps them forever (more on this in Part 4).

2. Scheduling a reminder with a deterministic job id

// src/services/orderReminders.ts
function reminderJobId(orderId: string, notificationNo: number): string {
  return `reminder:${orderId}:${notificationNo}`;
}

export async function scheduleReminder(orderId: string, notificationNo: number): Promise<string> {
  const jobId = reminderJobId(orderId, notificationNo);
  await addJob<ReminderJobData>(QUEUE, { orderId, notificationNo }, { jobId });
  await OrderModel.findByIdAndUpdate(orderId, { reminderJobId: jobId });
  return jobId;
}

Instead of letting BullMQ generate a random job id, we build one from the order id and the reminder number, for example reminder:6aaf...:2. Adding a job with an id that already exists does nothing, so the same reminder can never be scheduled twice. We also store the id on the order, so we can find and cancel the job later.

3. The worker: check, schedule the next reminder, then send

// src/services/orderReminders.ts
export async function processReminder(job: Job<ReminderJobData>): Promise<void> {
  const { orderId, notificationNo } = job.data;
  const order = await OrderModel.findById(orderId);

  // Stop if the order is gone or already paid.
  if (!order || order.status === 'paid') return;

  // Schedule the next reminder BEFORE sending this one.
  if (notificationNo < maxNotifications) {
    await scheduleReminder(orderId, notificationNo + 1);
  } else {
    await OrderModel.findByIdAndUpdate(orderId, { reminderJobId: null });
  }

  // If this throws, BullMQ retries it per `attempts` + `backoff`.
  await sendReminder(order.id, notificationNo);
}

Three things are happening here:

The check comes first. If the order was paid in the meantime, the job does nothing.
Each reminder schedules the next one, until maxNotifications is reached. This is how "remind up to 3 times" works: a chain of delayed jobs.
The next reminder is scheduled before sending. If sending fails on every retry, the next reminder is already in the queue, so one failed reminder doesn't stop the rest. And because the job id is deterministic, when BullMQ retries this job, scheduling the next reminder again is a no-op.

4. Cancelling on payment

// src/controllers/orderController.ts (payOrder)
// Mark paid first, so any reminder racing this request sees `paid` and no-ops.
order.status = 'paid';
await order.save();

// Then remove the one pending reminder — this ends the chain permanently.
// If Redis is unreachable this fails, but the payment is already saved and
// the worker's paid-status check stops the reminder when it runs, so the
// payment still succeeds.
try {
  await cancelReminder(order.id);
  order.reminderJobId = null;
} catch (err) {
  console.error('Could not cancel reminder (it will no-op when it runs):', (err as Error).message);
}

The order is marked paid first, then the pending reminder job is removed. Since each reminder schedules the next one, removing the single pending job ends the whole chain.

Cancelling is best-effort: if Redis happens to be down at that moment, the payment still succeeds. The reminder job stays in the queue, but when it runs, the check at the top of the worker sees the order is paid and does nothing.

The result, with an unpaid order and a paid one:

Reminder 1/3: order 6aaf3f79ecf4cf995f6ab3ed is still unpaid
Reminder 2/3: order 6aaf3f79ecf4cf995f6ab3ed is still unpaid
Reminder 3/3: order 6aaf3f79ecf4cf995f6ab3ed is still unpaid

The unpaid order got exactly 3 reminders; the paid order got none.

Before you go to production

1. Handle Redis outages. With the default settings, while Redis is down, every queue command waits in memory until Redis comes back, so your API requests just hang. The tempting fix is to cap the reconnect attempts so the client gives up, but then it never reconnects: after the outage, the app stays broken until you restart it. The right setup has three parts.

Keep reconnecting forever, with a growing (but capped) wait between attempts:

// src/config/config.ts
redis: {
  host: process.env.REDIS_HOST ?? '127.0.0.1',
  port: Number(process.env.REDIS_PORT ?? 6379),
  // Required by BullMQ for its blocking worker connection.
  maxRetriesPerRequest: null,
  // Keep reconnecting forever, waiting a little longer after each failed
  // attempt (200ms, 400ms, ...) up to REDIS_RETRY_DELAY_MS.
  retryStrategy: (times: number): number => Math.min(times * 200, redisMaxRetryDelayMs),
},

Make the queue (the producer) fail fast while Redis is down, instead of waiting:

// src/services/queue.ts
// enableOfflineQueue: false — while Redis is down, adding a job fails
// immediately instead of waiting in memory until Redis comes back, so the
// API can answer the client right away. (Workers keep the default.)
queue = new Queue(name, { connection: { ...config.redis, enableOfflineQueue: false } });

And at startup, wait for Redis only for a limited time, so the app exits instead of accepting requests it cannot serve:

// src/services/queue.ts
export async function verifyRedisConnection(timeoutMs: number): Promise<void> {
  let timer: NodeJS.Timeout | undefined;
  const timeout = new Promise<never>((_, reject) => {
    timer = setTimeout(
      () => reject(new Error(`Redis not reachable within ${timeoutMs}ms`)),
      timeoutMs,
    );
  });
  try {
    await Promise.race([getQueue(QUEUE_NAMES.ORDER_REMINDERS).waitUntilReady(), timeout]);
  } finally {
    clearTimeout(timer);
  }
}

With this, while Redis is down, requests fail in milliseconds; when it comes back, the API and the worker recover by themselves.

2. Don't leave half-done work behind. Creating an order is two steps: save the order, then schedule its reminder. If the second step fails (for example, Redis is down), remove the order and tell the client to retry. Otherwise the client sees an error, but an order that will never be reminded stays in your database:

// src/controllers/orderController.ts (createOrder)
const order = await OrderModel.create({ item, amount });

try {
  const jobId = await scheduleReminder(order.id, 1);
  order.reminderJobId = jobId;
} catch (err) {
  // The reminder could not be scheduled (e.g. Redis is down). Don't keep an
  // order that would never be reminded: remove it and tell the client to
  // retry, so an error response never leaves a hidden order behind.
  await OrderModel.deleteOne({ _id: order._id });
  console.error('Could not schedule reminder:', (err as Error).message);
  throw new AppError('could not schedule reminder, please retry', 503);
}

3. Add error listeners to every Queue and Worker. Without them, BullMQ prints a raw stack trace on every reconnect attempt while Redis is down, which floods your logs and hides everything else:

queue.on('error', (err) => console.error(`Queue ${name} connection error:`, err.message));
reminderWorker.on('error', (err) =>
  console.error('Reminder worker connection error:', err.message),
);

4. Run workers as a separate process. In the example app, the worker runs inside the API process to keep things simple. In production, run workers as their own process, so you can scale them independently and a slow job never blocks your API.

5. One worker per queue. A Worker processes exactly one queue. If you have several queues, create one Worker for each.

Full source code

The complete app (Express + MongoDB + BullMQ) is on GitHub: tarun-koshti/Redis-BullMq

To run it:

git clone https://github.com/tarun-koshti/Redis-BullMq.git
cd Redis-BullMq
docker compose up -d
npm install
cp .env.example .env
npm run dev

In the next part, we'll see what changes when you run Redis-backed jobs at a very large scale.

Part 4: Can I use it for very large applications?

Yes. Redis-backed job queues run in production at very large scale. But the approach changes as you grow. Small apps can rely on defaults; large apps need to plan capacity, partitioning, and durability deliberately.

1. Start by measuring jobs, not requests

Your request count and your job count are different numbers. An app handling 10,000 requests per second might create only a few hundred jobs per second if jobs come from failures, like webhook retries. Estimate your actual job rate first, and remember that each job costs many Redis operations over its lifecycle (adding, moving from delayed to ready, claiming, completing, plus lock renewals for long-running jobs). In our test with BullMQ 6, a single delayed job ran about 40 Redis commands, packed into just 2 Lua script calls.

2. Calculate your memory budget

A simple formula estimates how many jobs sit in Redis at once: job rate × average delay. At 10,000 jobs per second with a 15-minute delay, that's 9 million pending jobs. In our test, a job with a 1 KB payload took about 1.9 KB in Redis, so 9 million of them need roughly 17 GB of RAM just for the backlog.

Finished jobs count too: by default, BullMQ keeps every completed and failed job in Redis. Set removeOnComplete and removeOnFail so they don't pile up:

await queue.add('reminder', data, {
  delay: 30_000,
  removeOnComplete: 1000, // keep only the last 1,000 completed jobs
  removeOnFail: 5000, // keep more failed jobs, for debugging
});

3. Keep job payloads small

Store only a job ID and a few essential fields in Redis, and keep the full payload in your primary database or object storage. In our test, a job carrying just an order ID and a number took about 0.44 KB, over 4 times less than the 1 KB payload above. The bigger your payloads, the bigger the saving.

4. Scale horizontally by partitioning queues

Redis executes commands on a single main thread, so a bigger machine only helps so much. Instead, split work across multiple queues, for example by customer ID or event type, and spread them across instances or a Redis Cluster. Because queue libraries use multi-key operations, each individual queue must stay on a single shard. In BullMQ, you do this with a hash tag in the queue prefix (and the worker must use the same prefix):

// All keys of this queue share the {webhooks:shard3} hash tag,
// so Redis Cluster keeps the whole queue on one shard.
const queue = new Queue('webhooks', { connection, prefix: '{webhooks:shard3}' });

Many smaller queues scale; one giant queue doesn't.

5. Scale workers independently

Workers are stateless and can be added or removed freely. Watch the load they put on Redis, though: hundreds of workers polling aggressively create significant traffic on their own. Mature libraries avoid this with blocking commands and smarter polling, which is one more reason to prefer them over a hand-rolled scheduler at scale.

6. Plan for backlog spikes

The biggest danger at scale is not normal traffic but an outage somewhere else. If a major customer's webhook endpoint goes down, millions of retries can pile up in memory within hours. Protect yourself with per-destination rate limits, circuit breakers that pause delivery to failing endpoints, the noeviction memory policy, and alerts on memory usage and queue depth.

7. Strengthen durability to match the stakes

At large scale, even rare job loss affects many users. Start by checking persistence: the official Redis Docker image has the append-only file (AOF) turned off, and only saves snapshots now and then (after 1 hour if at least 1 key changed, after 5 minutes if 100 changed, after 1 minute if 10,000 changed). So a crash can lose several minutes of jobs, or up to an hour on a quiet instance. Turn on the AOF:

# redis.conf
appendonly yes
appendfsync everysec   # write to disk every second: a crash loses at most ~1 second

With Docker, the same is redis-server --appendonly yes as the container command.

Beyond that, options include WAITAOF (Redis 7.2+) to confirm writes are persisted before acknowledging them, durable managed services such as AWS MemoryDB, or keeping your primary database as the source of truth, with a periodic sweep to catch any jobs Redis lost.

8. Consider a hybrid architecture

Many large systems combine tools, each doing what it does best. A durable log like Kafka or a queue like SQS handles ingestion, a database stores job state, and Redis provides fast, precise timing. This gets you Redis's speed without depending on it alone for durability.

9. Invest in monitoring

At scale, you need visibility into queue depth, scheduling lag (how late jobs run compared to their scheduled time), memory usage, failure and dead-letter rates, and Redis operations per second. Scheduling lag is especially useful: it rises early when workers or Redis start falling behind.

Redis doesn't stop working at large scale, but it stops forgiving mistakes. With partitioned queues, small payloads, the right durability settings, and good monitoring, it can serve as the scheduling backbone of very large applications.

Wrapping up

In this article, we saw:

why Redis is a good fit for scheduling jobs (Part 1),
how to schedule, cancel and retry jobs with BullMQ (Part 2),
how this looks in a real Express app (Part 3),
and what changes at a very large scale (Part 4).

If you already run Redis, you can start with a single queue and grow from there.

The complete source code is on GitHub: tarun-koshti/Redis-BullMq

In the next article, we'll see how to use Redis as a cron service, to run jobs on a repeating schedule, like every night at 2 AM.

Thanks for reading!

AsyncLocalStorage in Node.js: stop passing requestId to every function

Tarun Koshti — Sun, 30 Aug 2026 07:31:48 +0000

Hello folks,

Recently I used a really interesting Node.js feature: AsyncLocalStorage.

What is AsyncLocalStorage?

AsyncLocalStorage lets you pass data across functions without passing it through function arguments. It ships with Node itself, in node:async_hooks, so there is nothing to install.

But wait — we can do the same thing with a global variable! So why do we need this?

Well, yes, you can do it with a global variable. But what if your app is serving 50 concurrent users? Let's say you store the request id in one:

User 1 arrives → gets request id U1 → their request takes 1 second to reach the next step
User 2 arrives in the meantime → a new request id is generated → assigned to that same global variable → now the rest of user 1's logs point at somebody else's request
Make that 50 concurrent users → 50 new request ids → tracing a single request becomes impossible

Global variables are fine for project-wide constants, but not for request-specific data.

Then what's the problem with passing it through function arguments?

Let's say you want a unique request identifier in your logs. You have 20 functions in the chain, and the only place that actually needs that identifier is the logger.

Most of those functions never log anything themselves — only the ones that make the HTTP call and write to the database do. But it is still your obligation to pass the request id through every downstream function, just so it can reach the one place that needs it.

That is where AsyncLocalStorage comes in. It carries the request id in its context, and whoever needs it simply reads it from the store.

Let's understand this with an example.

Example 1: passing requestId through every function argument

Notice how the request id has to be passed to every function on the way down.

// The "before" picture: requestId is threaded through every function argument.
// Count how many of these functions accept requestId without ever reading it.

// Shape of the JSON-ish payloads we pass around.
type TObject = Record<string, string | number | object | boolean>;

// Logger function.
// This is the ONLY function that actually reads requestId. Every other
// signature below just carries it along so that it can reach this one.
async function logger(logData: TObject, requestId: string) {
  const logObject = {
    requestId,
    request: logData,
  };
  console.log(JSON.stringify(logObject));
}

// Function to handle the request.
// The request lands here first, and this is where requestId is born.
async function handleRequest(requestData: TObject) {
  const requestId = crypto.randomUUID();

  // Example function to make an HTTP call
  const apiResult = await apiCall(requestData, requestId);

  // Example function to save the data in the database
  await saveData(apiResult, requestId);
}

// Function to make the HTTP request.
// requestId is dead weight here: it is forwarded to logger(), nothing more.
async function apiCall(requestData: TObject, requestId: string) {
  // Stand-in for a real API call: wait instead of hitting the network.
  await new Promise((resolve) => setTimeout(resolve, 200));

  // Dummy API response
  const response: TObject = {
    success: true,
    data: {
      userId: 'U1',
      paymentId: 'U_P1',
      sentPayload: requestData,
    },
  };

  await logger(response, requestId);
  return response;
}

// Function to save the data in the database.
// Same story: requestId is pure pass-through.
async function saveData(apiResult: TObject, requestId: string) {
  // Stand-in for a real database write.
  await new Promise((resolve) => setTimeout(resolve, 100));

  await logger({ saved: true, record: apiResult }, requestId);
}

// Run it:  npx tsx example.ts
await handleRequest({ userId: 'U1', amount: 4999 });

Here, just to get the request id into the logger, we had to pass it through 2 functions. Now imagine 20 functions instead — every single one of them has to accept the request id and forward it, and the moment one of them forgets, that log line loses its correlation.

Example 2: the same code with AsyncLocalStorage

Now it's time to see the real magic: how the request id travels without being passed through function arguments.

import { AsyncLocalStorage } from 'node:async_hooks';

interface Context {
  requestId: string;
}

const asyncLocalStorage = new AsyncLocalStorage<Context>();

// Read the current request's id out of the store. Exposing this accessor
// instead of the store itself keeps run() in one place.
function getRequestId(): string | undefined {
  return asyncLocalStorage.getStore()?.requestId;
}

// Shape of the JSON-ish payloads we pass around.
type TObject = Record<string, string | number | object | boolean>;

// Logger function.
// Only this function reads requestId, and it pulls it from the store instead
// of receiving it as an argument. Every other signature below stays clean.
async function logger(logData: TObject) {
  const requestId = getRequestId();
  const logObject = {
    requestId,
    request: logData,
  };
  console.log(JSON.stringify(logObject));
}

// Function to handle the request.
async function handleRequest(requestData: TObject) {
  // Example function to make an HTTP call
  const apiResult = await apiCall(requestData);

  // Example function to save the data in the database
  await saveData(apiResult);
}

// Function to make the HTTP request.
// It never takes or mentions requestId, yet its log line still carries one.
async function apiCall(requestData: TObject) {
  // Stand-in for a real API call: wait instead of hitting the network.
  await new Promise((resolve) => setTimeout(resolve, 200));

  // Dummy API response
  const response: TObject = {
    success: true,
    data: {
      userId: 'U1',
      paymentId: 'U_P1',
      sentPayload: requestData,
    },
  };

  await logger(response);
  return response;
}

// Function to save the data in the database.
// Same story: a clean signature, and a correlated log line anyway.
async function saveData(apiResult: TObject) {
  // Stand-in for a real database write.
  await new Promise((resolve) => setTimeout(resolve, 100));

  await logger({ saved: true, record: apiResult });
}

// The requestId is created once, here at the boundary. Everything called
// inside run() can read it: across awaits, through nested calls, all the way
// down to logger().
asyncLocalStorage.run(
  {
    requestId: crypto.randomUUID(),
  },
  async () => {
    await handleRequest({ userId: 'u1', paymentId: 'U_P1' });
  },
);

Here you can see we don't pass anything through the functions. We store the requestId once in AsyncLocalStorage, and the logger picks it up at the final logging step.

Both log lines come out carrying the same id:

{"requestId":"84762d3d-5f73-428b-9995-29f9b34a477a","request":{"success":true,"data":{...}}}
{"requestId":"84762d3d-5f73-428b-9995-29f9b34a477a","request":{"saved":true,"record":{...}}}

Compare the two signatures side by side, and that is the whole argument:

before:  logger(logData, requestId)   apiCall(requestData, requestId)   saveData(apiResult, requestId)
after:   logger(logData)              apiCall(requestData)              saveData(apiResult)

Example 3: the same thing in Express, using express-http-context

Both examples above were plain scripts. In a real app the boundary is your HTTP layer, and there is a small package that wires exactly this up for you: express-http-context.

It is worth knowing what it actually is before you reach for it. It has no dependencies, and under the hood it is the very same AsyncLocalStorage we just used by hand — its middleware is literally asyncLocalStorage.run(new Map(), () => next()). So this is not an alternative approach, it is the same approach with the plumbing already written.

npm install express express-http-context

// The same example as before, now wired into Express with express-http-context.
//
// In the AsyncLocalStorage version we wrote three pieces of plumbing by hand:
//   1. the store     ->  new AsyncLocalStorage<Context>()
//   2. the boundary  ->  asyncLocalStorage.run({ requestId }, ...)
//   3. the accessor  ->  getRequestId()
//
// express-http-context takes the place of all three. It is a small wrapper
// with no dependencies, and under the hood it is the very same
// AsyncLocalStorage -- it just holds the store as a Map<string, unknown>, so
// you address values by key instead of through a typed Context object.

import express from 'express';
import httpContext from 'express-http-context';
import { randomUUID } from 'node:crypto';

const app = express();
app.use(express.json());

// 1. Create the per-request store.
//    This one line replaces both `new AsyncLocalStorage()` and the
//    `asyncLocalStorage.run(...)` wrapper from the previous example.
//    Internally the middleware is literally:
//        asyncLocalStorage.run(new Map(), () => next())
//    Because next() is called inside run(), every middleware, route and
//    function further down the chain inherits the same store.
app.use(httpContext.middleware);

// 2. Put the request id into that store, once, at the boundary.
//    Previously this was the object we handed to run({ requestId: ... }).
//    Careful: set() silently does nothing when the middleware above is
//    missing, so there is no error to tell you the context was never opened.
app.use((req, res, next) => {
  httpContext.set('requestId', req.header('x-request-id') ?? randomUUID());
  next();
});

// Logger function.
// As we saw above, only this function reads requestId, and it pulls it from
// the store instead of receiving it as an argument. Every other signature
// below stays clean.
// The one change from the previous example: our custom getRequestId() helper
// is gone, and httpContext.get('requestId') now does that job for us.
async function logger(logData) {
  const requestId = httpContext.get('requestId');
  const logObject = {
    requestId,
    request: logData,
  };
  console.log(JSON.stringify(logObject));
}

// Function to make the HTTP request.
// Unchanged from the AsyncLocalStorage example: it never takes or mentions
// requestId, yet its log line still carries one.
async function apiCall(requestData) {
  // Stand-in for a real API call: wait instead of hitting the network.
  await new Promise((resolve) => setTimeout(resolve, 200));

  // Dummy API response
  const response = {
    success: true,
    data: {
      userId: 'U1',
      paymentId: 'U_P1',
      sentPayload: requestData,
    },
  };

  // Log. Nothing about the request id is passed in.
  await logger(response);
  return response;
}

// Function to save the data in the database.
// Also unchanged: a clean signature, and a correlated log line anyway.
async function saveData(apiResult) {
  // Stand-in for a real database write.
  await new Promise((resolve) => setTimeout(resolve, 100));

  // Log again, still without passing the request id anywhere.
  await logger({ saved: true, record: apiResult });
}

// This route takes the place of handleRequest() from the previous example.
// Notice there is no run() call to wrap it in: httpContext.middleware already
// opened the context for this request, so everything called from here can
// read the id.
app.post('/orders', async (req, res) => {
  // Example function to make an HTTP call
  const apiResult = await apiCall(req.body);

  // Example function to save the data in the database
  await saveData(apiResult);

  // Reading the id here too, to hand it back to the caller -- handy in a real
  // app, since the client can then quote it in a bug report.
  res.status(201).json({ success: true, requestId: httpContext.get('requestId') });
});

// Fire two requests at the same time and watch the ids stay separate:
//   curl -X POST localhost:3000/orders -H 'content-type: application/json' -d '{"userId":"u1"}'
app.listen(3000, () => {
  console.log('listening on http://localhost:3000');
});

Run it with node, then fire two requests at once. The log lines interleave, but each keeps its own id:

{"requestId":"04757118-...","request":{"success":true,"data":{"sentPayload":{"userId":"u2",...}}}}
{"requestId":"cdce047b-...","request":{"success":true,"data":{"sentPayload":{"userId":"u1",...}}}}
{"requestId":"04757118-...","request":{"saved":true,"record":{...u2...}}}
{"requestId":"cdce047b-...","request":{"saved":true,"record":{...u1...}}}

Two things to watch out for with this package specifically:

set() fails silently if you forget app.use(httpContext.middleware), or if you call it outside a request. Nothing is stored and no error is raised.
The store is a Map<string, unknown>, so httpContext.get('requestId') gives you any. In a TypeScript codebase the hand-rolled AsyncLocalStorage<Context> version keeps its types, which is a good reason to stay with the built-in.

A few things to keep in mind

AsyncLocalStorage is not magic, and the context does not follow your data everywhere:

It does not survive a queue. Work you push onto a job queue during a request and run later has no context. Capture it with AsyncLocalStorage.snapshot() at enqueue time.
EventEmitter listeners inherit the context of whoever calls .emit(), not the one they were registered in. Wrap them with AsyncResource.bind().
A shared or memoised promise belongs to whoever created it. If two requests await the same in-flight promise, its body runs in the first request's context — which can silently mix up more than just logs.
Prefer run() over enterWith(). enterWith() has no boundary, so it leaks back into the caller.
There is a small cost. Activating AsyncLocalStorage turns on promise hooks for the whole process, so every promise gets a little slower. For a request that already spends milliseconds in I/O that is noise; in a hot loop, read the store once at the top and pass the value down from there.

Wrapping up

If you are threading a request id, a tenant id, a trace id or a logged-in user through function signatures that don't care about any of it, AsyncLocalStorage removes that parameter entirely. Your service and repository functions go back to taking only the arguments they actually use, and your logs stay correlated anyway.

It is built into Node, needs no dependency, and in a real app the whole integration is usually one middleware.

I have put a full working example with Express in this repo — the same API written both ways, plus runnable demos for each of the gotchas above:

https://github.com/tarun-koshti/Node-Async-Local-Storage-Example

If you have used AsyncLocalStorage in production, I would love to hear where it helped you, or where it bit you. Drop a comment.

Thanks for reading!