BullMQ is easy to start with. You create a Queue, a Worker and a Redis connection, and jobs flow. Most of the problems show up later, under load, during a deploy or when Redis restarts.
This post covers common BullMQ mistakes, most of them called out in the official docs, why each one happens, and how to fix it. Everything here is checked against the BullMQ source (v6.4.2) and the official going-to-production guide. At the end there's a lint config that catches four of them before you ship.
1. Sharing one ioredis connection with maxRetriesPerRequest left at the default
This is the most common one. (In BullMQ 6, ioredis is an optional peer dependency, so install both: npm i bullmq ioredis.)
import { Queue, Worker } from 'bullmq';
import IORedis from 'ioredis';
const connection = new IORedis(process.env.REDIS_URL);
const queue = new Queue('emails', { connection });
const worker = new Worker('emails', processEmail, { connection });
A Worker sits in blocking Redis commands while it waits for jobs. ioredis has a per-command retry limit, maxRetriesPerRequest, which defaults to a finite number. If Redis drops for a few seconds, a blocking command can hit that limit and fail instead of waiting for the reconnect.
BullMQ handles this two different ways depending on what you pass:
- If you pass connection options, BullMQ sets
maxRetriesPerRequest: nullfor workers and logs a warning if you set something else. - If you pass an ioredis instance, it can't change the instance's options, so the
Workerconstructor throwsBullMQ: Your redis options maxRetriesPerRequest must be null.
The fix is to give workers their own connection with the right setting:
const workerConnection = new IORedis(process.env.REDIS_URL, { maxRetriesPerRequest: null });
const worker = new Worker('emails', processEmail, { connection: workerConnection });
Producers are the opposite case. An API route that calls queue.add() should fail fast if Redis is down, so the client gets an error instead of a request that hangs. Keep a finite maxRetriesPerRequest on the connection your API uses.
2. No 'error' listener
const worker = new Worker('emails', processEmail, { connection });
// no worker.on('error', ...)
Worker and Queue are Node.js event emitters. When an emitter emits 'error' and nobody is listening, Node throws, which is how a short Redis blip becomes a crashed process. The going-to-production guide says to attach an error handler to both:
worker.on('error', (err) => logger.error({ err }, 'worker error'));
queue.on('error', (err) => logger.error({ err }, 'queue error'));
This is separate from the 'failed' event, which fires when your processor throws for a specific job.
3. Assuming jobs retry by default
They don't. A job with no attempts option runs once, and if the processor throws, it goes straight to failed. From BullMQ's retry docs: "you should use the attempts setting with a value larger than 1."
Set it once as a queue default, with backoff, so a flaky downstream API doesn't get hammered:
const queue = new Queue('emails', {
connection,
defaultJobOptions: {
attempts: 5,
backoff: { type: 'exponential', delay: 1000 },
removeOnComplete: { age: 24 * 3600, count: 1000 },
removeOnFail: { age: 7 * 24 * 3600, count: 5000 },
},
});
attempts counts the total number of runs, not the number of retries. For errors that will never succeed on retry (bad input, a deleted user), throw BullMQ's UnrecoverableError so the job fails immediately instead of using up its attempts.
The removeOnComplete and removeOnFail settings are there for a different reason. Without them, every finished job stays in Redis forever.
4. Breaking jobId deduplication
A custom jobId is BullMQ's simplest way to avoid duplicate jobs. If a webhook gets delivered twice, you add the job with the event id, and the second add() is ignored. Two things commonly break it.
First, the id format. BullMQ uses : as its Redis key separator, and in the v6.4 source, Job throws Custom Id cannot contain : for most ids with a colon, and Custom Id cannot be integers for ids that are integer strings like '42':
await queue.add('welcome', data, { jobId: `user:${id}` }); // throws
await queue.add('welcome', data, { jobId: String(id) }); // throws if id is numeric
await queue.add('welcome', data, { jobId: `user-${id}` }); // fine
These throw at runtime, often in a code path your tests don't cover.
Second, deduplication only works while the original job still exists. This combination quietly turns it off:
await queue.add('charge', data, { jobId: `evt-${event.id}`, removeOnComplete: true });
removeOnComplete: true deletes the job as soon as it finishes. When the same event is delivered again five minutes later, there's no job with that id any more, so BullMQ adds a new one and the work runs twice. Keep completed jobs for at least your dedupe window, with removeOnComplete: { age: <seconds> }, or use BullMQ's deduplication: { id, ttl } option, which is built for this.
5. Stalled jobs from a blocked event loop
When a worker picks up a job, it takes a lock in Redis and renews it while the job runs. The defaults in the v6.4 Worker constructor are a 30-second lockDuration and a 30-second stalledInterval. If the lock isn't renewed in time, BullMQ treats the job as stalled and moves it back to waiting so another worker can run it. With the default maxStalledCount of 1, a job that stalls a second time fails with "job stalled more than allowable limit."
There are two common causes:
- CPU-bound work in the processor. The stalled jobs guide explains that a processor which keeps the CPU busy blocks the event loop, and the lock can't be renewed. The fix is to break the work up, or use a sandboxed processor (a file path instead of a function) so it runs in a separate process.
-
Killing workers during deploys. If the process is killed mid-job, the lock expires and the job stalls. Listen for
SIGTERMandSIGINTand callworker.close(), which stops fetching new jobs and waits for active ones to finish.
for (const sig of ['SIGINT', 'SIGTERM']) {
process.once(sig, async () => {
await worker.close();
await workerConnection.quit();
process.exit(0);
});
}
A stalled job runs again, so processors need to be safe to run twice. That's a good rule for queue workers anyway.
One Redis setting: maxmemory-policy
This isn't in your code, but the guide calls it out: set Redis maxmemory-policy to noeviction. With any eviction policy, Redis can delete queue keys when it runs low on memory, and jobs disappear. If you use a managed Redis, check what policy your plan sets.
Catching these with a linter
Four of these mistakes can be found by reading the code, so I wrote them up as ESLint rules in eslint-plugin-bullmq:
| Rule | Catches |
|---|---|
worker-connection-max-retries |
an ioredis connection used by a Worker or QueueEvents without maxRetriesPerRequest: null (mistake 1) |
require-error-listener |
a Worker or Queue with no 'error' listener (mistake 2) |
valid-job-id |
a literal or template jobId containing :, or an integer (mistake 4) |
no-jobid-dedupe-with-remove-on-complete |
jobId combined with removeOnComplete: true, including through defaultJobOptions (mistake 4) |
Setup with ESLint's flat config:
npm i -D eslint eslint-plugin-bullmq
// eslint.config.js
import bullmq from 'eslint-plugin-bullmq';
export default [bullmq.configs.recommended];
Running it against this file:
// worker.js
import { Queue, Worker } from 'bullmq';
import IORedis from 'ioredis';
const connection = new IORedis(process.env.REDIS_URL);
const queue = new Queue('emails', { connection });
queue.on('error', console.error);
const worker = new Worker('emails', async () => {}, { connection });
export async function signup(id) {
await queue.add('welcome', { id }, { jobId: `user:${id}`, removeOnComplete: true });
}
gives this (messages shortened):
5:20 error This ioredis connection is used by a BullMQ Worker but does not set maxRetriesPerRequest: null ... bullmq/worker-connection-max-retries
8:16 error This Worker has no 'error' listener ... bullmq/require-error-listener
11:47 error BullMQ uses ':' as its Redis key separator, so a custom jobId containing ':' is rejected ... bullmq/valid-job-id
11:61 warning removeOnComplete: true deletes the job as soon as it finishes ... bullmq/no-jobid-dedupe-with-remove-on-complete
Two rules come with suggestions you can apply from the editor, adding maxRetriesPerRequest: null and replacing : with -. For TypeScript files, add it after typescript-eslint's config so ESLint picks up .ts files.
The linter can't catch mistakes 3 and 5, since a missing attempts might be deliberate and a blocked event loop only shows at runtime.
Starting from a template instead
If you're starting a new service, create-bullmq-app scaffolds an API and a separate worker with these fixes already in place. It has separate Redis options for producers (maxRetriesPerRequest: 3) and workers (null), 'error' listeners on every worker, attempts with exponential backoff and bounded removeOnComplete/removeOnFail as queue defaults, and graceful shutdown on SIGINT/SIGTERM. The generated project has eslint-plugin-bullmq turned on, so it starts out passing these rules.
npm create bullmq-app@latest my-service
Checklist
- Workers get a connection with
maxRetriesPerRequest: null. The API connection stays finite. - Every
WorkerandQueuehas an'error'listener. -
attemptsandbackoffare set, plusremoveOnCompleteandremoveOnFailwith limits. - Custom job ids have no
:and aren't plain integers, and completed jobs live at least as long as your dedupe window. - Processors don't block the event loop, and workers close on
SIGTERM. - Redis runs with
maxmemory-policy noeviction.
Disclosure: I'm Krrish Kumar, and I maintain eslint-plugin-bullmq and create-bullmq-app.
Top comments (0)