A Complete Guide with Workers, Rate Limiting, Retries & Dead-Letter Queues (DLQs) 🚀🔥
When your NestJS application grows, running cron jobs becomes… tricky.
Running multiple instances means your cron jobs will run multiple times, which is usually very bad 😅.
To avoid duplicate jobs and chaos, we’ll build a robust distributed cron system using:
- NestJS
- BullMQ (modern queue system)
- Redis
- Worker processes
- Retry logic
- Dead-letter queues (DLQs)
- Rate-limiting
And all wrapped in a multi-instance setup using Docker Compose.
Let’s go. 🛠️✨
🧱 Architecture Overview
We will build two separate NestJS applications:
┌────────────────────┐
│ Scheduler Service │ ← runs cron jobs
│ (NestJS App #1) │
└──────────┬─────────┘
│ pushes jobs
▼
┌─────────────┐
│ BullMQ │
│ (Redis) │
└──────┬──────┘
│ workers pull jobs
▼
┌────────────────────┐
│ Worker Service │ ← processes jobs
│ (NestJS App #2) │
└────────────────────┘
- Only the Scheduler runs cron jobs
- Only the Worker processes jobs
- Both communicate through Redis
- All instances stay synchronized
- No duplicated tasks 🎉
🧰 Step 1 — Install Redis
The easiest method is Docker:
docker run -d --name redis -p 6379:6379 redis:7
📦 Step 2 — Install Dependencies
Install BullMQ & NestJS integration:
npm i bullmq @nestjs/bullmq
npm i -D @types/bull
Install scheduling support:
npm i @nestjs/schedule
(Optional) install Bull Dashboard:
npm i @bull-board/api @bull-board/express
🔌 Step 3 — Create a Shared Redis Config
Create src/bull-config.ts:
export const redisConfig = {
connection: {
host: process.env.REDIS_HOST ?? 'localhost',
port: Number(process.env.REDIS_PORT ?? 6379),
},
};
You’ll reuse this in both apps.
🕒 Step 4 — Build the Scheduler App
(The app that runs cron jobs and pushes them into queues)
Enable ScheduleModule:
// scheduler/app.module.ts
@Module({
imports: [
ScheduleModule.forRoot(),
BullModule.forRoot(redisConfig),
BullModule.registerQueue({
name: 'email_queue',
}),
],
})
export class AppModule {}
Create a Cron Job
// scheduler/email-cron.service.ts
import { Cron, CronExpression } from '@nestjs/schedule';
import { InjectQueue } from '@nestjs/bullmq';
import { Queue } from 'bullmq';
@Injectable()
export class EmailCronService {
constructor(@InjectQueue('email_queue') private emailQueue: Queue) {}
@Cron(CronExpression.EVERY_10_SECONDS)
async handleCron() {
console.log('⏰ Adding job to email queue...');
await this.emailQueue.add(
'send-welcome-email',
{ userId: Math.floor(Math.random() * 1000) },
{
attempts: 3,
backoff: { type: 'exponential', delay: 3000 },
removeOnComplete: true,
removeOnFail: false,
rateLimiter: {
max: 5,
duration: 1000,
},
}
);
}
}
What we added:
- 🔄 Retries (3 attempts)
- ⏳ Exponential backoff
- 🚦 Rate-limiting (5 jobs per second)
- 🧹 Cleanup completed jobs
🛠️ Step 5 — Build the Worker App
(This app consumes & executes queue jobs)
// worker/app.module.ts
@Module({
imports: [
BullModule.forRoot(redisConfig),
BullModule.registerQueue({
name: 'email_queue',
}),
],
providers: [EmailProcessor],
})
export class AppModule {}
Create the Job Processor
// worker/email.processor.ts
import { Processor, WorkerHost } from '@nestjs/bullmq';
import { Job } from 'bullmq';
@Processor('email_queue')
export class EmailProcessor extends WorkerHost {
async process(job: Job<any>): Promise<any> {
try {
console.log(`📨 Processing email job for User ${job.data.userId}`);
if (Math.random() < 0.3) {
throw new Error('Simulated email failure!');
}
return { success: true };
} catch (err) {
console.error('❌ Job failed:', err);
throw err;
}
}
}
💀 Step 6 — Dead-Letter Queue (DLQ)
Create a secondary queue called email_queue:dlq.
Modify worker:
@Processor('email_queue')
export class EmailProcessor extends WorkerHost {
async onFailed(job: Job, error: any) {
const dlq = new Queue('email_dlq', redisConfig);
await dlq.add('dlq-job', {
failedJob: job.id,
data: job.data,
error: error.message,
});
console.log(`💀 Moved job ${job.id} to DLQ`);
}
}
Now any job that fails after max retries goes into the dead-letter queue,
ready for manual review or reprocessing.
🧱 Step 7 — Dockerize Everything
Here is the full working docker-compose.yml:
version: "3.9"
services:
redis:
image: redis:7
ports:
- "6379:6379"
scheduler:
build: ./scheduler
depends_on:
- redis
environment:
REDIS_HOST: redis
deploy:
replicas: 1
worker:
build: ./worker
depends_on:
- redis
environment:
REDIS_HOST: redis
deploy:
replicas: 3
What this achieves:
- Scheduler has 1 replica → cron runs once
- Worker can have as many replicas as needed → job processing scales horizontally
📊 Optional: Add BullMQ Dashboard
Inside the worker app:
import { createBullBoard } from '@bull-board/api';
import { BullMQAdapter } from '@bull-board/api/bullMQAdapter';
import { ExpressAdapter } from '@bull-board/express';
@Module({})
export class DashboardModule implements OnModuleInit {
onModuleInit() {
const serverAdapter = new ExpressAdapter();
const emailQueue = new Queue('email_queue', redisConfig);
createBullBoard({
queues: [new BullMQAdapter(emailQueue)],
serverAdapter,
});
serverAdapter.setBasePath('/dashboard');
app.use('/dashboard', serverAdapter.getRouter());
}
}
Open dashboard:
http://localhost:3000/dashboard
🎉 Final Result: A Rock-Solid Cron Job System
You now have:
✅ A Scheduler app that runs cron jobs
✅ A Worker app that processes tasks
❌ No duplicated cron executions
🚦 Rate-limiting
🔁 Retry logic with exponential backoff
💀 Dead-letter queues
🐳 Horizontal scaling with Docker
🎛️ Optional admin dashboard
All production-ready.
All beautifully decoupled.
All powered by Redis & BullMQ.
Top comments (2)
This definitely does not solve the problems in the conclusion section: multiple instance will introduce one job in the queue each, geneating N jobs executions. In particular:
Hi ivanva,
Thanks for the comment, and you're absolutely right, so I corrected the article and added a couple of extras.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.