Background Job Queues with Claude Code: BullMQ Patterns for Node.js

#claudecode #node #redis #typescript

Email sending, image processing, report generation — these should never block your HTTP response thread. Claude Code can design a complete BullMQ queue system when given the right patterns.

CLAUDE.md for Job Queue Standards

## Job Queue Rules

### When to use queues (required)
- Any operation taking > 3 seconds
- External API calls (email, payments, webhooks)
- File processing (image resize, PDF generation)
- Batch operations (reports, data exports)

### Library: BullMQ (Redis Streams-based, TypeScript native)

### Design rules
- Job data type required (define JobData interface)
- Retry: max 3 attempts, exponential backoff
- Timeout: set per job (default: 30 seconds)
- removeOnComplete: true (don't accumulate data)
- removeOnFail: 100 (keep for debugging)

### Naming convention
- {domain}-{action}: `email-send`, `image-resize`, `report-generate`

### Monitoring
- Bull Board dashboard at /admin/queues
- Alert when failure rate > 5%

Generating the BullMQ System

Generate a job queue system using BullMQ.

Job type: email sending (email-send queue)
Job data: { to: string, subject: string, body: string, userId: string }
Retry: max 3 attempts, exponential backoff (1000ms * 2^retryCount)
Timeout: 30 seconds
TypeScript

Generate these files:
- src/queues/emailQueue.ts (queue definition)
- src/workers/emailWorker.ts (worker process)
- src/queues/types.ts (job data interfaces)

Generated Code

// src/queues/types.ts
export interface EmailJobData {
  to: string;
  subject: string;
  body: string;
  userId: string;
}

// src/queues/emailQueue.ts
import { Queue } from 'bullmq';
import { EmailJobData } from './types';

export const emailQueue = new Queue<EmailJobData>('email-send', {
  connection: redis,
  defaultJobOptions: {
    attempts: 3,
    backoff: { type: 'exponential', delay: 1000 },
    removeOnComplete: true,
    removeOnFail: 100,
  },
});

export async function enqueueEmail(data: EmailJobData) {
  return emailQueue.add('send', data, { timeout: 30_000 });
}

// src/workers/emailWorker.ts
import { Worker, Job } from 'bullmq';

export const emailWorker = new Worker<EmailJobData>(
  'email-send',
  async (job: Job<EmailJobData>) => {
    const { to, subject, body, userId } = job.data;
    logger.info({ userId, to, jobId: job.id }, 'Sending email');
    await sendEmail({ to, subject, html: body });
    logger.info({ userId, to, jobId: job.id }, 'Email sent');
  },
  { connection: redis, concurrency: 5 }
);

emailWorker.on('failed', (job, err) => {
  logger.error(
    { jobId: job?.id, userId: job?.data.userId, error: err.message },
    'Email job failed'
  );
});

Job Progress Tracking API

Generate a job progress tracking API.

Requirements:
- POST /jobs/export — create job, return jobId
- GET /jobs/:jobId/status — return { status, progress, downloadUrl? }
  status: 'waiting' | 'active' | 'completed' | 'failed'
- Worker calls job.updateProgress(0-100) for progress updates
- Return download URL when completed

Implement in: src/routes/jobs.ts

Generated routes:

router.post('/export', async (req, res) => {
  const job = await reportQueue.add('generate', {
    userId: req.user.id,
    filters: req.body.filters,
  });
  res.json({ jobId: job.id });
});

router.get('/:jobId/status', async (req, res) => {
  const job = await reportQueue.getJob(req.params.jobId);
  if (!job) return res.status(404).json({ error: 'Job not found' });

  const state = await job.getState();
  res.json({
    status: state,
    progress: job.progress,
    ...(state === 'completed' && {
      downloadUrl: `/downloads/${job.returnvalue}`,
    }),
    ...(state === 'failed' && { error: job.failedReason }),
  });
});

Adding Bull Board Dashboard

Add Bull Board web UI to Express app.

Requirements:
- Path: /admin/queues
- Protected by Basic Auth (ADMIN_USER/ADMIN_PASS env vars)
- Monitor queues: email-send, image-resize, report-generate