DEV Community: Bar-Dov

Avoiding Redis Crashes with BullMQ: Memory Monitoring Basics

Bar-Dov — Mon, 14 Jul 2025 09:15:00 +0000

If you’re using BullMQ with Redis in production, you’ve probably dealt with a "wtf happened" moment after Redis hit its memory limit. 🧨😐

We hit this once (on a Sunday, naturally). Since then, we started tracking Redis memory usage per instance and get notified when we hit ~80% of capacity.

💡 Turns out some Redis hosts don’t expose their max memory, so we set a manual threshold in MB. Helps us catch issues way before Redis shuts the door on writes.

We later baked this into our own job monitoring tool. If you want to keep things simple and focused on BullMQ, we made Upqueue.io to alert on stuff like memory spikes, stuck queues, and missing workers.

The Max Memory monitors page looks like this (see below)

But tool or not—do monitor your Redis memory. Especially if you’re chaining long-running jobs or dealing with a backlog.

What are you all using? Custom scripts? Hosted dashboards?

How We Handle BullMQ Queue Backlogs (Without Stressing Over It)

Bar-Dov — Sun, 06 Jul 2025 12:11:07 +0000

One of the most overlooked issues when working with BullMQ (or any job queue, really) is quietly building backlogs. It doesn’t crash the system—but it slows things down until something breaks.

After a few rough incidents (hello weekend alerts 😅), we started tracking how many jobs are waiting, prioritized, or delayed per queue. If the number crosses a certain threshold, we get alerted and scale up.

🔧 Pro tip: If you’re not monitoring queue backlog yet, set a soft threshold based on your infra. Even just 100 pending jobs on a medium-sized queue might be enough to indicate a bottleneck.

We ended up building a minimal dashboard just for this—eventually turned it into Upqueue.io. Might help if you’re looking for something more lightweight than full observability platforms. No config hell, just watch your queues.

Below you can see the page where we can easily set backlog alerts

Would love to hear how others handle backlog monitoring. Manual scripts? Prometheus? No clue and vibes?

BullMQ UI: Why Bull Board May Not Be Enough (And How Upqueue.io Helps)

Bar-Dov — Wed, 25 Jun 2025 15:56:16 +0000

When your queue looks “quiet,” it might actually be failing silently

TL;DR

Bull Board is a solid starting point with basic job visibility.
Upqueue.io is built for production observability: alerts, metrics, child-job support, and UI polish make a difference.

If you're running BullMQ in production, you’ve likely got jobs, workers, and retries configured. But all the functionality in the world won’t help if you can’t see what’s actually happening.

That’s where a true BullMQ UI becomes essential.

A Good BullMQ UI Should Offer

✅ Live status (active, delayed, completed, failed)
✅ Job retry/delete controls
✅ Job context with JSON/logs
✅ Historical trends for queues
✅ Alerts for failures, stalled queues, memory/connection issues
✅ Visibility into child (nested) jobs

Let’s compare Bull Board — the go‑to starter UI — with Upqueue.io, which is built for observability-first production use.

Feature Comparison: Bull Board vs Upqueue.io

Quick Setup Example


// queue.ts
import { Queue, Worker } from 'bullmq';

export const reportQueue = new Queue('report', {
  connection: { host: 'localhost', port: 6379 },
});

new Worker('report', async job => {
  return await generatePDF(job.data);
});

All good—but to run safely, you need a UI that shows when something breaks.

Upqueue.io connects to your Redis and instantly provides:

✅ Failed Jobs monitor
✅ Connection & Memory alerts
✅ Missing Workers tracking
✅ Backlog alerts
✅ Child‑job tab and retry controls
✅ Clean, developer-friendly interface

Oh, and bonus—a real team behind it, with fast support and new features dropping regularly.

Learn more and explore the dashboard: upqueue.io

Using BullMQ to Power AI Workflows (with Observability in Mind)

Bar-Dov — Sun, 08 Jun 2025 10:29:38 +0000

As AI-based applications become more sophisticated, managing their asynchronous tasks becomes increasingly complex. Whether you’re generating content, processing embeddings, or chaining together multiple model calls—queues are essential infrastructure.

And for many Node.js applications, BullMQ has become the go-to queueing library.

In this post, we’ll walk through why BullMQ fits well into AI pipelines, and how to handle some of the pitfalls that come with running critical async work at scale.

Why BullMQ Makes Sense for AI Workflows

AI jobs are often:

CPU/GPU intensive (model inference)
Long running (fine-tuning, summarizing large chunks)
Chainable (one output feeds the next)
Best handled asynchronously

Queues help break down these processes into manageable, distributed units.

Example: A Simple AI Pipeline with BullMQ

Let’s say you’re building a summarization service.

User submits a document.

The job is queued.

A worker generates the summary.

A follow-up task sends it via email.

Here’s how you might structure that with BullMQ:

// queues.ts
import { Queue } from 'bullmq';
import { connection } from './redis-conn';

export const summarizationQueue = new Queue('summarize', { connection });
export const emailQueue = new Queue('email', { connection });

// producer.ts
await summarizationQueue.add('summarizeDoc', {
  docId: 'abc123',
});

// summarization.worker.ts
import { Worker } from 'bullmq';
import { summarizationQueue, emailQueue } from './queues';

new Worker('summarize', async job => {
  const summary = await generateSummary(job.data.docId);

  await emailQueue.add('sendEmail', {
    userId: job.data.userId,
    summary,
  });
});

You can imagine how this might expand:

Queue for transcription
Queue for sentiment analysis
Queue for search index updates

What to Watch Out For

When you're handling large numbers of AI jobs:

Memory usage spikes can crash your Redis instance.

Worker failures can leave queues silently stuck.

Job retries without proper limits can pile up fast.

These are hard to track without some sort of observability layer.

Good Practices for AI Queue Systems

✅ Use job removeOnComplete: true to avoid memory buildup
✅ Set attempts and backoff on your long-running jobs
✅ Monitor failed jobs & queue lengths
✅ Alert on missing workers or high backlog

Even a minimal dashboard that shows which queues are stuck or which workers are down can save hours.

We had to build one ourselves. If you’re looking for something simple and focused, we put together a tool called Upqueue.io that visualizes BullMQ jobs and alerts you when things go wrong. But whether it’s a custom script, Prometheus, or something else - just make sure you’re not flying blind.

BullMQ is a great fit for AI apps. But the more you scale, the more you need to see what’s going on.

Don’t let your GPT worker crash at 3am without you knowing.

Monitor early. Sleep better.