BullMQ Has a Free API — The Ultimate Node.js Job Queue

#node #javascript #backend #redis

BullMQ is the most popular job queue for Node.js — built on Redis, with support for delayed jobs, rate limiting, job priorities, and real-time progress tracking. Completely free and open source.

Why BullMQ?

Redis-backed — reliable, fast, battle-tested storage
Job priorities — process critical jobs first
Rate limiting — control throughput per queue
Delayed jobs — schedule execution for later
Repeatable jobs — cron-like scheduling
Progress tracking — real-time job progress updates
Concurrency control — process N jobs in parallel
Retries with backoff — exponential or custom retry logic

Quick Start

npm install bullmq
# Redis must be running (docker run -d -p 6379:6379 redis)

Producer (Add Jobs)

import { Queue } from "bullmq";

const emailQueue = new Queue("emails", {
  connection: { host: "localhost", port: 6379 },
});

// Simple job
await emailQueue.add("welcome", {
  to: "user@example.com",
  subject: "Welcome!",
  template: "welcome",
});

// Delayed job (send in 1 hour)
await emailQueue.add("reminder", {
  to: "user@example.com",
  subject: "Don't forget to complete your profile",
}, {
  delay: 60 * 60 * 1000, // 1 hour
});

// Priority job (processed first)
await emailQueue.add("password-reset", {
  to: "user@example.com",
  subject: "Password Reset",
}, {
  priority: 1, // Lower number = higher priority
});

// Repeatable job (every day at 9am)
await emailQueue.add("daily-digest", {
  template: "digest",
}, {
  repeat: { cron: "0 9 * * *" },
});

console.log("Jobs added to queue");

Worker (Process Jobs)

import { Worker } from "bullmq";

const worker = new Worker("emails", async (job) => {
  console.log(`Processing ${job.name}: ${job.data.to}`);

  // Update progress
  await job.updateProgress(10);

  // Simulate sending email
  const result = await sendEmail(job.data);

  await job.updateProgress(100);

  return { sent: true, messageId: result.id };
}, {
  connection: { host: "localhost", port: 6379 },
  concurrency: 5, // Process 5 emails at a time
  limiter: {
    max: 100,      // Max 100 emails
    duration: 60000, // Per minute (rate limit)
  },
});

worker.on("completed", (job, result) => {
  console.log(`Job ${job.id} completed:`, result);
});

worker.on("failed", (job, err) => {
  console.error(`Job ${job.id} failed:`, err.message);
});

worker.on("progress", (job, progress) => {
  console.log(`Job ${job.id}: ${progress}%`);
});

Flow (Job Dependencies)

import { FlowProducer } from "bullmq";

const flowProducer = new FlowProducer({
  connection: { host: "localhost", port: 6379 },
});

// Parent job waits for all children to complete
await flowProducer.add({
  name: "process-order",
  queueName: "orders",
  data: { orderId: "ORD-001" },
  children: [
    {
      name: "validate-payment",
      queueName: "payments",
      data: { amount: 99.99 },
    },
    {
      name: "check-inventory",
      queueName: "inventory",
      data: { items: ["widget-1", "gadget-2"] },
    },
    {
      name: "calculate-shipping",
      queueName: "shipping",
      data: { address: "123 Main St" },
    },
  ],
});

// "process-order" only runs after all 3 children complete

Real-World: Image Processing Pipeline

import { Queue, Worker } from "bullmq";

const imageQueue = new Queue("images", {
  connection: { host: "localhost", port: 6379 },
  defaultJobOptions: {
    attempts: 3,
    backoff: { type: "exponential", delay: 2000 },
    removeOnComplete: { count: 1000 },
    removeOnFail: { count: 5000 },
  },
});

// Add image processing job
await imageQueue.add("resize", {
  userId: "U-123",
  imageUrl: "https://example.com/photo.jpg",
  sizes: [
    { width: 1200, height: 630, name: "og" },
    { width: 400, height: 400, name: "thumbnail" },
    { width: 100, height: 100, name: "avatar" },
  ],
});

// Worker
const worker = new Worker("images", async (job) => {
  const { imageUrl, sizes } = job.data;
  const results = [];

  for (let i = 0; i < sizes.length; i++) {
    const size = sizes[i];
    await job.updateProgress(Math.round((i / sizes.length) * 100));

    // Process image (using sharp, jimp, etc.)
    const outputPath = await resizeImage(imageUrl, size);
    results.push({ ...size, path: outputPath });
  }

  await job.updateProgress(100);
  return { processed: results.length, files: results };
}, {
  connection: { host: "localhost", port: 6379 },
  concurrency: 3,
});

Dashboard (Bull Board — Free)

import { createBullBoard } from "@bull-board/api";
import { BullMQAdapter } from "@bull-board/api/bullMQAdapter";
import { ExpressAdapter } from "@bull-board/express";
import express from "express";

const serverAdapter = new ExpressAdapter();
serverAdapter.setBasePath("/admin/queues");

createBullBoard({
  queues: [
    new BullMQAdapter(emailQueue),
    new BullMQAdapter(imageQueue),
  ],
  serverAdapter,
});

const app = express();
app.use("/admin/queues", serverAdapter.getRouter());
app.listen(3000, () => console.log("Dashboard: http://localhost:3000/admin/queues"));

BullMQ vs Celery vs SQS vs Bee-Queue

Feature	BullMQ	Celery	AWS SQS	Bee-Queue
Language	Node.js	Python	Any	Node.js
Backend	Redis	Redis/RabbitMQ	AWS	Redis
Job flows	Yes	Canvas	Step Functions	No
Rate limiting	Built-in	Built-in	No	No
Dashboard	Bull Board	Flower	CloudWatch	No
Priority	Yes	Yes	No	Yes
Repeatable	Built-in	Celery Beat	EventBridge	No

Need to scrape data from any website and get it in structured JSON? Check out my web scraping tools on Apify — no coding required, results in minutes.

Have a custom data extraction project? Email me at spinov001@gmail.com — I build tailored scraping solutions for businesses.

DEV Community