Building Resilient Background Jobs in NestJS with BullMQ

Background jobs look simple right up until one of them dies silently in production and nobody notices for three days. A job that sends confirmation emails stops running. A job that syncs inventory data quietly falls behind. Nobody gets an error, because from the queue's perspective, nothing "crashed" — the job just failed and nobody was watching.

Most BullMQ tutorials stop at "job added, job processed." That's fine for a demo, but it's not what happens in a real system. In production, external APIs time out, workers restart mid-job, and retries without the right safeguards can make things worse, not better.

In this post, I'll skip the basic setup tutorial and goes straight into the patterns that actually matter: retries that don't cause a thundering herd, idempotency so retries don't duplicate side effects, dead-letter queues for jobs that keep failing, concurrency limits that protect your database, and how to catch jobs that are "done" but still stuck.

Quick Setup

If you haven't wired up BullMQ in a NestJS app yet, here's the minimum you need.

npm install @nestjs/bullmq bullmq ioredis

// app.module.ts
import { Module } from '@nestjs/common';
import { BullModule } from '@nestjs/bullmq';

@Module({
  imports: [
    BullModule.forRoot({
      connection: {
        host: process.env.REDIS_HOST,
        port: Number(process.env.REDIS_PORT),
      },
    }),
    BullModule.registerQueue({
      name: 'notifications',
    }),
  ],
})
export class AppModule {}

// notifications.processor.ts
import { Processor, WorkerHost } from '@nestjs/bullmq';
import { Job } from 'bullmq';

@Processor('notifications')
export class NotificationsProcessor extends WorkerHost {
  async process(job: Job): Promise<void> {
    // send the notification
  }
}

That's the happy path. Now let's make it survive contact with production.

Retries with Exponential Backoff

The default instinct is to just add retries and move on:

await notificationsQueue.add('send-email', payload, {
  attempts: 5,
});

The problem: without a backoff strategy, BullMQ retries as fast as it can. If the reason the job failed was a downstream API having a bad moment, five instant retries from every failed job in the queue can turn a blip into an outage — a thundering herd hitting a service that's already struggling.

Exponential backoff spaces retries out so the downstream system gets room to recover:

await notificationsQueue.add('send-email', payload, {
  attempts: 5,
  backoff: {
    type: 'exponential',
    delay: 1000, // 1s, 2s, 4s, 8s, 16s
  },
});

For queues with high volume, add jitter on top of this. If a downstream outage causes thousands of jobs to fail at the same moment, exponential backoff alone still retries them all in near-lockstep — you've just delayed the thundering herd, not prevented it. A custom backoff strategy fixes that:

BullModule.registerQueue({
  name: 'notifications',
  defaultJobOptions: {
    backoff: {
      type: 'custom',
    },
  },
});

// custom-backoff.strategy.ts
export function customBackoffStrategy(attemptsMade: number): number {
  const base = Math.min(1000 * 2 ** attemptsMade, 30000);
  const jitter = Math.random() * 0.3 * base;
  return base + jitter;
}

Register it on the worker's settings.backoffStrategy option so retries spread out instead of clustering.

Idempotency: The Part Most Tutorials Skip

Retries assume it's safe to run the job again. That assumption breaks constantly. A job that charges a customer, sends an email, or writes to an external system can cause real damage if it runs twice — the first attempt may have actually succeeded downstream even though your worker crashed before it could report success back to BullMQ.

The fix is an idempotency key: a unique identifier for "this specific unit of work," checked before the job runs its side effects.

// idempotency.service.ts
import { Injectable } from '@nestjs/common';
import { PrismaService } from '../prisma/prisma.service';

@Injectable()
export class IdempotencyService {
  constructor(private prisma: PrismaService) {}

  async hasRun(key: string): Promise<boolean> {
    const record = await this.prisma.idempotencyKey.findUnique({
      where: { key },
    });
    return !!record;
  }

  async markComplete(key: string): Promise<void> {
    await this.prisma.idempotencyKey.create({ data: { key } });
  }
}

// notifications.processor.ts
@Processor('notifications')
export class NotificationsProcessor extends WorkerHost {
  constructor(private idempotency: IdempotencyService) {
    super();
  }

  async process(job: Job): Promise<void> {
    const key = `send-email:${job.data.userId}:${job.data.templateId}`;

    if (await this.idempotency.hasRun(key)) {
      return; // already sent, retry is a no-op
    }

    await sendEmail(job.data);
    await this.idempotency.markComplete(key);
  }
}

This pattern costs one extra database round trip per job, which is a fair trade against duplicate charges or duplicate emails landing in a customer's inbox.

Dead-Letter Queues for Jobs That Keep Failing

Once a job exhausts its attempts, BullMQ marks it failed and moves on. If nothing is watching for that event, the job's failure disappears into the queue's history — it won't page anyone, and it won't show up unless someone happens to check.

A dead-letter queue (DLQ) gives failed jobs a place to land where they're visible and replayable.

// dead-letter.listener.ts
import { OnQueueEvent, QueueEventsListener, QueueEventsHost } from '@nestjs/bullmq';
import { InjectQueue } from '@nestjs/bullmq';
import { Queue } from 'bullmq';

@QueueEventsListener('notifications')
export class NotificationsDeadLetterListener extends QueueEventsHost {
  constructor(@InjectQueue('notifications-dlq') private dlq: Queue) {
    super();
  }

  @OnQueueEvent('failed')
  async onFailed({ jobId, failedReason }: { jobId: string; failedReason: string }) {
    const job = await this.getJob(jobId);

    if (job && job.attemptsMade >= job.opts.attempts!) {
      await this.dlq.add('failed-notification', {
        originalData: job.data,
        failedReason,
        failedAt: new Date().toISOString(),
      });
    }
  }
}

From there, a small replay script lets you re-queue jobs from the DLQ once the underlying issue is fixed:

// replay-dlq.script.ts
async function replayDlq(dlq: Queue, notifications: Queue) {
  const jobs = await dlq.getJobs(['waiting', 'delayed']);

  for (const job of jobs) {
    await notifications.add('send-email', job.data.originalData);
    await job.remove();
  }
}

The DLQ doesn't fix the underlying failure — it turns "silently lost job" into "visible, replayable job," which is the difference between a five-minute fix and a customer complaint three days later.

Concurrency Control and Backpressure

It's tempting to crank up concurrency to clear a backlog faster:

@Processor('notifications', { concurrency: 50 })
export class NotificationsProcessor extends WorkerHost {
  // ...
}

Fifty concurrent jobs sounds fine until each one opens a database connection, and your Postgres pool only has 20 connections available. The queue isn't the bottleneck anymore — the database is, and now every other part of the app competes with the job queue for connections.

Size concurrency around your actual downstream constraints, not an arbitrary number:

@Processor('notifications', {
  concurrency: 10, // matched to available DB pool headroom
})
export class NotificationsProcessor extends WorkerHost {
  // ...
}

If the queue talks to a rate-limited third-party API, BullMQ's built-in limiter is a better fit than concurrency alone:

BullModule.registerQueue({
  name: 'notifications',
  limiter: {
    max: 100,
    duration: 60000, // max 100 jobs per minute
  },
});

Detecting Stuck or Silently Hanging Jobs

The failure mode nobody plans for: a job that's technically still "active" from BullMQ's perspective, but is actually hung — stuck waiting on a call to an external service that never times out and never resolves. It won't show up as failed. It'll just sit there, occupying a worker slot indefinitely.

BullMQ's stalled-job detection catches part of this — if a worker dies mid-job without renewing its lock, the job gets marked stalled and retried:

BullModule.registerQueue({
  name: 'notifications',
  settings: {
    stalledInterval: 30000,
    maxStalledCount: 2,
  },
});

That handles a crashed worker. It doesn't handle a worker that's still alive but stuck in a call with no timeout. For that, the job itself needs a hard timeout:

async process(job: Job): Promise<void> {
  await Promise.race([
    sendEmail(job.data),
    new Promise((_, reject) =>
      setTimeout(() => reject(new Error('Job exceeded timeout')), 15000),
    ),
  ]);
}

Without a timeout like this, "the job is still running" and "the job is stuck forever" look identical from the outside. This is usually where a team ends up building or buying observability into their job queues — once you've been paged for a queue that quietly backed up for six hours, you stop trusting "it'll show up as failed eventually."

Wrapping Up

None of these patterns are exotic — retries, idempotency, dead-letter queues, concurrency limits, and timeouts are well-known concepts. What's easy to miss is that BullMQ gives you the primitives, not the judgment calls: how long to back off, whether a job is safe to retry, what "stuck" actually means for your workload. Those decisions are what separate a queue that works in a demo from one that survives production traffic.

If you're setting up background jobs in NestJS for the first time, start with idempotency and a DLQ before you worry about tuning concurrency — those two alone catch the failure modes that actually wake people up at night.

Comments

[Nazar Boyko]

The idempotency check might want the write to happen first. As written, hasRun passes, the email goes out, and if the worker dies before markComplete the retry sends it again, which is exactly the crash you describe at the top of that section. Inserting the key up front with a unique constraint and treating the duplicate error as "someone already claimed this" also handles two workers hitting the same key at once.