Monitoring Python RQ jobs: what to watch and how to get alerted

RQ (Redis Queue) is a delightfully simple way to run background jobs in Python. That simplicity is also why teams under-monitor it: it just works, until a downstream API gets slow or a bad deploy ships, and jobs start failing in bulk — quietly. Here's what to watch and how to get alerted before a customer tells you.

RQ failures don't announce themselves

When a job raises, RQ moves it to the FailedJobRegistry and moves on. The worker keeps running; nothing crashes. If you're not looking at that registry, the failure is invisible — the same trap BullMQ, Celery, and every robust queue share. So the job is to reach into the queue's state and turn it into a signal.

The four signals that matter for RQ

1. Failure count / rate — jobs landing in the FailedJobRegistry over a window.
2. Backlog — how many jobs are queued vs. being worked; is the worker keeping up?
3. Latency — how long jobs take, and how long they wait before a worker picks them up.
4. Worker liveness — are your workers actually alive and heartbeating?

Where to read them

RQ exposes queue and registry state directly:

from redis import Redis
from rq import Queue
from rq.registry import FailedJobRegistry, StartedJobRegistry

redis = Redis()
q = Queue("default", connection=redis)

queued   = len(q)                                   # backlog
failed   = FailedJobRegistry(queue=q)               # failures
started  = StartedJobRegistry(queue=q)              # in-flight

print("queued:", queued)
print("failed:", len(failed))
print("started:", len(started))

Poll this on an interval and store the series — a single snapshot hides the trend, which is the part that matters. For failures specifically, walk the registry to get the actual exceptions:

for job_id in failed.get_job_ids():
    job = q.fetch_job(job_id)
    print(job.id, job.exc_info.splitlines()[-1] if job.exc_info else "")

Two gotchas:

• Group by exception, not by job. A thousand jobs failing with the same traceback is one incident. Normalize the message (strip IDs, timestamps, hosts) and group on the rest, or you'll drown in near-identical entries.
• Watch worker heartbeats. RQ workers register in Redis; if a worker dies mid-job the job can sit in StartedJobRegistry past its TTL. A rising started-but-never-finished count is a worker-health problem masquerading as a queue problem.

Turning signals into alerts

The metrics above are only useful if something evaluates them and pages a human. Options:

• A cron + your own thresholds — cheap, but you build the rate math, grouping, and delivery.
• Prometheus + Grafana — export the registry counts (e.g. rq-exporter) and alert in Grafana.
• A hosted monitor — send events out and let it handle windows, grouping, and routing.

The principle is identical across every queue: watch from outside the worker, alert on the rate (not a single failure), and group identical failures so a retry storm is one page.

Where PipeRadar fits (and doesn't, yet)

Full disclosure: I work on PipeRadar, a hosted monitor that does exactly this
— failure clustering, latency percentiles, history, and rate-based alerts to Slack/PagerDuty/webhooks
— for BullMQ today. RQ is on the roadmap, not shipping yet, so I'm not going to pretend you can pip install it this afternoon.

What's relevant now: PipeRadar's ingest API is queue-agnostic (it already models an adapter type per queue system), so an RQ adapter is the missing piece, not a rewrite. If RQ monitoring like this is something you'd use, the fastest way to pull it up the roadmap is to say so — you can follow progress and register interest at piperadar.dev. In the meantime, the patterns above work with nothing but Redis and a cron.

---

More on background-job monitoring — including a deep dive on
why queue jobs fail silently (BullMQ, but the failure model is the same in RQ) — at piperadar.dev/blog.