DEV Community

ZNY
ZNY

Posted on

The Complete Guide to Background Job Processing in 2026: Bull, Bee, and Graphile Workers

The Complete Guide to Background Job Processing in 2026: Bull, Bee, and Graphile Workers

Email sending, image processing, webhooks, report generation — none of these belong in your HTTP request cycle.

Why Background Jobs

Synchronous processing in HTTP handlers:

  • User waits for your heavy computation to finish
  • Connection drops = job fails with no retry
  • Can't scale workers independently from web servers

Background jobs solve all of this.

BullMQ — The Redis-Backed Queue 

import { Queue, Worker } from 'bullmq';
import Redis from 'ioredis';

const connection = new Redis({ maxRetriesPerRequest: null });

// Producer
const emailQueue = new Queue('emails', { connection });
await emailQueue.add('welcome', {
  to: 'user@example.com',
  template: 'welcome'
}, {
  attempts: 3,
  backoff: { type: 'exponential', delay: 1000 }
});

// Worker
const worker = new Worker('emails', async (job) => {
  console.log(`Processing job ${job.id}`);
  await sendEmail(job.data);
}, { connection });

worker.on('failed', (job, err) => {
  console.error(`Job ${job?.id} failed:`, err.message);
});
Enter fullscreen mode Exit fullscreen mode

Scheduled Jobs (Cron) 

await emailQueue.add(
  'digest',
  { userId: 123 },
  {
    repeat: {
      pattern: '0 9 * * *', // Daily at 9 AM
      tz: 'America/New_York'
    }
  }
);
Enter fullscreen mode Exit fullscreen mode

Job Priorities and Concurrency 

const worker = new Worker('orders', async (job) => {
  // Process order
}, {
  connection,
  concurrency: 10,           // Process 10 jobs simultaneously
  limiter: {
    max: 50,
    duration: 1000           // Max 50 per second
  }
});

// Priority: lower number = higher priority
await orderQueue.add('create', data, { priority: 1 });
await orderQueue.add('notify', data, { priority: 5 });
Enter fullscreen mode Exit fullscreen mode

Graphile Worker — Simpler Alternative 

// tasks/notification.ts
import { task } from '@datapartytown/graphile-worker';

export const sendNotification = task(async (payload) => {
  const { userId, message } = payload;
  await notifyUser(userId, message);
});
Enter fullscreen mode Exit fullscreen mode 
# Run worker
npx graphile-worker -c ./worker.sql
Enter fullscreen mode Exit fullscreen mode

Choosing the Right Queue

Feature BullMQ Graphile Worker AWS SQS
Persistence Redis PostgreSQL AWS managed
UI Dashboard Optional Built-in CloudWatch
Priorities ✅ ✅ ❌
Scheduled Jobs ✅ ✅ ✅
Job Retry ✅ ✅ ✅
Cluster Mode ✅ ❌ ✅

Error Handling Patterns 

// Dead letter queue for failed jobs
worker.on('failed', async (job, err) => {
  if (job.attemptsMade >= job.opts.attempts) {
    await deadLetterQueue.add('failed-job', {
      originalQueue: job.queueName,
      jobId: job.id,
      data: job.data,
      error: err.message,
      failedAt: new Date().toISOString()
    });
  }
});
Enter fullscreen mode Exit fullscreen mode

Conclusion

Background jobs are essential for any real application. BullMQ is the most flexible, Graphile Worker is the simplest to operate. Start with whatever your team can maintain.

Focus on your application logic, not infrastructure — deploy workers alongside your app with automatic scaling and zero configuration.

Top comments (0)