DEV Community: Harsh

Debugging Stalled Jobs in BullMQ: What Actually Causes Them and How to Catch Them Early published: false tags: node, redis, bullmq, backend

Harsh — Wed, 08 Jul 2026 09:25:45 +0000

If you've run BullMQ in production long enough, you've probably seen a job go "stalled" — and if you haven't, you will. Here's what actually causes it, why it's more dangerous than a simple failure, and how to catch it before it costs you.

What "stalled" actually means

A stalled job isn't the same as a failed job. A failed job threw an error and BullMQ knows about it. A stalled job is one BullMQ thinks is still being processed, but the worker that picked it up has gone silent — no heartbeat, no completion, nothing. From BullMQ's perspective, the job is in limbo.

This usually happens because of:

Worker crash mid-job — an unhandled exception, OOM kill, or container restart while a job is actively processing
Event loop blocking — a long synchronous operation (heavy computation, a blocking file read, a bad regex) prevents the worker from sending its lock-renewal heartbeat in time
Lock expiration under load — if your lockDuration is too short relative to how long jobs actually take, jobs can be marked stalled even when the worker is still healthily processing them
Redis connectivity blips — a brief network partition between the worker and Redis can cause the lock renewal to fail even though the worker itself is fine

Why stalled jobs are more dangerous than failures

A failed job is visible — it's in your failed queue, it has an error message, you can see it and retry it. A stalled job just... sits there, silently, until BullMQ's stalled-check interval notices it and either retries it or marks it failed, depending on your config. In the meantime, if that job was part of a time-sensitive flow (sending a confirmation email, processing a payment webhook, updating inventory), nothing tells you it's stuck. You find out when a customer complains, not when it happens.

How to reduce stalled jobs

A few concrete things that help:

Tune lockDuration and stalledInterval to match your actual job runtime, not a guess. If your jobs typically take 30 seconds, a 30-second lock is too tight — give yourself real margin.
Avoid blocking the event loop inside job processors. Move CPU-heavy work to worker threads or a separate process if you can. Node's single-threaded event loop means a bad synchronous chunk of code can silently prevent heartbeats from firing.
Set maxStalledCount deliberately. The default retry behavior for stalled jobs can mask a recurring problem if you don't watch for jobs that stall repeatedly — that's usually a sign of a systemic issue (e.g. a specific job type that's too slow), not bad luck.
Log job start and heartbeat events, not just completion and failure, so you have a timeline to look at after the fact.

Catching it before it becomes an incident

The hardest part of stalled jobs isn't fixing them once you know — it's finding out in time. BullMQ emits events for this (stalled, failed, active, completed) via its QueueEvents class, but most teams either don't wire up alerting on these events at all, or only find out by checking the dashboard reactively.

This is actually the problem we built Qcanary to solve — it's a small agent that listens to these BullMQ events locally in your own worker process and sends an alert (Slack, email, webhook) the moment something stalls or fails repeatedly, without needing access to your Redis instance directly. If you're currently finding out about stalled jobs from a customer instead of an alert, that gap is worth closing one way or another, whether that's this or building your own listener on QueueEvents.

Takeaway

Stalled jobs are a symptom, not a root cause — they almost always point to either a lock/timing mismatch or blocking code in your processor. Fix the underlying cause where you can, but also make sure you find out when it happens instead of waiting for it to surface downstream. A silent stall in a critical path is far more expensive than a loud failure.

What's the worst stalled-job incident you've dealt with? Curious to hear how other teams have caught (or missed) these in production.

A practical guide to monitoring BullMQ queues with an agent-based approach that keeps Redis credentials inside your infrastructure.

Harsh — Sun, 05 Jul 2026 16:09:58 +0000

devopscover_image: https://qcanary.dev/screenshots/dashboard-overview.png

Background jobs are the invisible backbone of your Node.js app. When they fail, they fail silently.

BullMQ is a great default for Node.js background jobs. It's fast, Redis-backed, and flexible enough for emails, reports, webhooks, and retry-heavy workflows. But once BullMQ is in production, a critical question shows up quickly:

How do you monitor queues without giving another service direct access to Redis?

Many hosted queue dashboards ask for your Redis URL. That works, but it comes with heavy security tradeoffs:

🔑 Credential exposure. Your production Redis URL gets stored by a third party.
🔥 Firewall changes. You have to open port 6379 or set up VPC peering—just for a dashboard.
📋 Compliance surface. SOC 2 and zero-trust policies require a vendor risk assessment for that access.
🗄️ Full data access. Redis has no row-level security. Once someone has the URL, they have everything.

This post walks through an alternative: using BullMQ's built-in QueueEvents emitter to monitor queues without ever sharing a Redis credential.
The Problem With Direct Redis Monitoring

A traditional hosted queue monitor connects directly to Redis and inspects queue state. That's convenient for building dashboards, but uncomfortable for production systems.

Depending on your Redis permissions and deployment model, a Redis connection can expose far more than queue metrics — keys, job state, retry metadata, delayed jobs, and sometimes payload data. Even when a vendor is trustworthy, the credential becomes another secret to audit, rotate, and protect.

The better model is simple:

Keep Redis private. Send only the monitoring events you actually need.

How BullMQ QueueEvents Changes the Game

BullMQ emits lifecycle events as jobs move through a queue. QueueEvents is the BullMQ primitive for subscribing to those changes inside your own process:

completed → failed → active → waiting → stalled → delayed → drained

Instead of polling Redis from a hosted service, you can subscribe from inside your runtime and translate these lifecycle events into monitoring records.

This eliminates:

 ✅ Inbound network access to Redis
 ✅ Redis credential sharing with vendors
 ✅ The need to inspect arbitrary Redis keys
 ✅ Firewall changes and VPC peering

Monitoring data is buffered locally and sent over standard outbound HTTPS.
The Agent Pattern in Practice

Here's how it works in a real application. First, install the agent:
bash

npm install @qcanary/agent

Then attach it to your existing BullMQ queues:
typescript

import { Queue } from "bullmq";
import { QueueMonitor } from "@qcanary/agent";

const emailQueue = new Queue("email", {
connection: { host: "127.0.0.1", port: 6379 },
});

const monitor = new QueueMonitor({
apiKey: process.env.QCANARY_API_KEY,
queues: [emailQueue],
});

await monitor.start();

The agent does a few things under the hood:

Attaches to QueueEvents as a local subscriber — your Redis connection never leaves the process.
Buffers events into batches (100 events or 5 seconds, whichever comes first).
Ships only lightweight metadata over HTTPS: job ID, queue name, status, duration, error message.
Retries with exponential backoff on failure.
Drops oldest events if the buffer fills — monitoring should never slow down your workers.

No job payloads. No Redis keys. No connection strings. Just events.
Setting Up Your First Alert

Dashboards are useful. Alerts are what make monitoring operational.

A good first alert is failure rate:
text

Queue: email
Condition: failure rate > 5%
Window: 10 minutes
Destination: Slack

One failed job may be noise. A sustained 5% failure rate over 10 minutes usually means something changed: a provider outage, expired credentials, a bad deploy, or a slow dependency.

Other useful alerts to configure:

 No activity — catch queues that silently stopped receiving work.
 Queue depth — catch growing backlogs before users feel the delay.
 Job duration — catch slow jobs and external dependency regressions.

Comparing the Approaches
Aspect

Sharing Redis URL

Agent-Based (QueueEvents)
Credential exposure Redis URL stored by vendor None — Redis stays local
Network changes Open port 6379 or VPC peering None — outbound HTTPS only
Compliance impact Vendor risk assessment required None — no shared infrastructure
Data sent Direct Redis access (everything) Only lifecycle metadata
Payload visibility Vendor can read all job data Only metadata — payloads stay local
Setup time 10 minutes + network config 5 minutes, no network config

What to Monitor First

Start with high-impact queues:

 Email delivery
 Billing webhooks
 User notifications
 Scheduled reports
 Data imports and exports
 Third-party sync jobs

Keep thresholds conservative at first. After a week of history, tune them based on normal production behavior.
What's Next for QCanary? (Expanding the Ecosystem)

Right now, QCanary is built specifically for BullMQ because of its clean event model. But we know the Node.js ecosystem is diverse.

We are actively expanding our agent to support the original Bull library, so legacy Node.js applications can secure their monitoring without migrating their queue infrastructure. Furthermore, we are laying the groundwork to support queue systems in other languages, bringing this same zero-trust, agent-based observability to Python, Go, and beyond.

If you're using Bull or another queue system, we'd love to hear from you!
When Sharing Redis Still Makes Sense

The agent pattern isn't always the right call:

 Local development — credential sharing doesn't matter on localhost.
 Full payload inspection needed — if you need to see job data in the dashboard, the agent can't help (it intentionally excludes payloads).
 Existing VPC peering — if the network boundary is already established, sharing Redis adds no new attack surface.

For everyone else — production deployments with VPCs, compliance requirements, or zero-trust policies — the agent pattern eliminates an entire class of security risk.
Try It Free

QCanary provides real-time dashboards, Slack/email/webhook alerts, and job history for BullMQ — all without ever seeing your Redis URL.

Free tier: 1 project, 3 queues, 10K events/day, 3-day history.
Setup: Install the agent, attach to your queues, events start flowing.

→ qcanary.dev
→ qcanary.dev/docs

Found this useful? Follow me for more Node.js infrastructure content. Questions? Drop them in the comments below.