Background Jobs in Node.js: BullMQ, Cron, and Worker Threads

#node #webdev #backend #typescript

Background Jobs in Node.js: BullMQ, Cron, and Worker Threads

Some work shouldn't block an HTTP response. Background jobs handle it asynchronously.

When to Use Background Jobs

Sending emails (user shouldn't wait for SMTP)
Processing uploads (resize, convert, store)
Generating reports (can take minutes)
Syncing data with external services
Anything that takes > 500ms

BullMQ: Production Queue

npm install bullmq ioredis

import { Queue, Worker } from 'bullmq';
import { Redis } from 'ioredis';

const connection = new Redis(process.env.REDIS_URL!);

// Queue definition
const emailQueue = new Queue('emails', { connection });

// Add job to queue (in your API route)
await emailQueue.add('welcome-email', {
  userId: user.id,
  email: user.email,
  name: user.name,
}, {
  attempts: 3,
  backoff: { type: 'exponential', delay: 2000 },
  removeOnComplete: true,
  removeOnFail: false, // Keep failed jobs for debugging
});

// Worker (separate process)
const worker = new Worker('emails', async (job) => {
  const { userId, email, name } = job.data;

  await resend.emails.send({
    from: 'hello@yourapp.com',
    to: email,
    subject: 'Welcome!',
    html: welcomeEmailTemplate(name),
  });

  console.log(`Welcome email sent to ${email}`);
}, { connection, concurrency: 5 });

worker.on('failed', (job, err) => {
  console.error(`Job ${job?.id} failed:`, err);
});

Cron Jobs

// Schedule repeating jobs
await emailQueue.add(
  'weekly-digest',
  { type: 'weekly-digest' },
  {
    repeat: { cron: '0 9 * * 1' }, // Every Monday at 9 AM
    jobId: 'weekly-digest', // Prevent duplicates
  }
);

Job Progress

// Worker: report progress
const worker = new Worker('reports', async (job) => {
  await job.updateProgress(10);
  const data = await fetchData();

  await job.updateProgress(50);
  const report = await generateReport(data);

  await job.updateProgress(90);
  await uploadToS3(report);

  await job.updateProgress(100);
  return { reportUrl: '...' };
}, { connection });

// Client: poll progress
const job = await reportQueue.getJob(jobId);
const progress = await job?.progress; // 0-100

Bull Board: Visual Dashboard

import { createBullBoard } from '@bull-board/api';
import { BullMQAdapter } from '@bull-board/api/bullMQAdapter';
import { ExpressAdapter } from '@bull-board/express';

const serverAdapter = new ExpressAdapter();
createBullBoard({
  queues: [new BullMQAdapter(emailQueue), new BullMQAdapter(reportQueue)],
  serverAdapter,
});

app.use('/admin/queues', serverAdapter.getRouter());
// Visual dashboard at /admin/queues