If you've run BullMQ in production long enough, you've probably seen a job go "stalled" — and if you haven't, you will. Here's what actually causes it, why it's more dangerous than a simple failure, and how to catch it before it costs you.
What "stalled" actually means
A stalled job isn't the same as a failed job. A failed job threw an error and BullMQ knows about it. A stalled job is one BullMQ thinks is still being processed, but the worker that picked it up has gone silent — no heartbeat, no completion, nothing. From BullMQ's perspective, the job is in limbo.
This usually happens because of:
Worker crash mid-job — an unhandled exception, OOM kill, or container restart while a job is actively processing
Event loop blocking — a long synchronous operation (heavy computation, a blocking file read, a bad regex) prevents the worker from sending its lock-renewal heartbeat in time
Lock expiration under load — if your lockDuration is too short relative to how long jobs actually take, jobs can be marked stalled even when the worker is still healthily processing them
Redis connectivity blips — a brief network partition between the worker and Redis can cause the lock renewal to fail even though the worker itself is fine
Why stalled jobs are more dangerous than failures
A failed job is visible — it's in your failed queue, it has an error message, you can see it and retry it. A stalled job just... sits there, silently, until BullMQ's stalled-check interval notices it and either retries it or marks it failed, depending on your config. In the meantime, if that job was part of a time-sensitive flow (sending a confirmation email, processing a payment webhook, updating inventory), nothing tells you it's stuck. You find out when a customer complains, not when it happens.
How to reduce stalled jobs
A few concrete things that help:
Tune lockDuration and stalledInterval to match your actual job runtime, not a guess. If your jobs typically take 30 seconds, a 30-second lock is too tight — give yourself real margin.
Avoid blocking the event loop inside job processors. Move CPU-heavy work to worker threads or a separate process if you can. Node's single-threaded event loop means a bad synchronous chunk of code can silently prevent heartbeats from firing.
Set maxStalledCount deliberately. The default retry behavior for stalled jobs can mask a recurring problem if you don't watch for jobs that stall repeatedly — that's usually a sign of a systemic issue (e.g. a specific job type that's too slow), not bad luck.
Log job start and heartbeat events, not just completion and failure, so you have a timeline to look at after the fact.
Catching it before it becomes an incident
The hardest part of stalled jobs isn't fixing them once you know — it's finding out in time. BullMQ emits events for this (stalled, failed, active, completed) via its QueueEvents class, but most teams either don't wire up alerting on these events at all, or only find out by checking the dashboard reactively.
This is actually the problem we built Qcanary to solve — it's a small agent that listens to these BullMQ events locally in your own worker process and sends an alert (Slack, email, webhook) the moment something stalls or fails repeatedly, without needing access to your Redis instance directly. If you're currently finding out about stalled jobs from a customer instead of an alert, that gap is worth closing one way or another, whether that's this or building your own listener on QueueEvents.
Takeaway
Stalled jobs are a symptom, not a root cause — they almost always point to either a lock/timing mismatch or blocking code in your processor. Fix the underlying cause where you can, but also make sure you find out when it happens instead of waiting for it to surface downstream. A silent stall in a critical path is far more expensive than a loud failure.
What's the worst stalled-job incident you've dealt with? Curious to hear how other teams have caught (or missed) these in production.
Top comments (0)