A queue that's silently falling behind looks identical to a healthy one until someone notices the emails are three hours late. Monitoring is what closes that gap, and you don't need an enterprise observability contract to get real visibility into queue depth, worker throughput, and failure rate. Here are five free tools worth setting up before your queue is the thing paging you at 2am.
1. Prometheus
Prometheus is the default choice for metrics collection in most modern infrastructure, and for good reason: it pulls metrics on a schedule, stores them as time series, and lets you query them with PromQL for exactly the kind of question a queue raises constantly, like "what's the 95th percentile job processing time over the last hour." Most queue libraries either expose a Prometheus endpoint natively or have a community exporter that does. Once queue depth, job duration, and failure counts are flowing into Prometheus, everything downstream (dashboards, alerts) gets much easier to build.
2. Grafana
Metrics without a dashboard are just numbers nobody looks at until something breaks. Grafana turns Prometheus data into the kind of dashboard you can glance at during a standup: queue depth over time, per-worker throughput, retry rate by job type. The free tier covers everything a small-to-mid-sized team needs, and prebuilt dashboard templates exist for most popular queue backends, so you're rarely starting from a blank panel.
3. OpenTelemetry
Metrics tell you that something is slow. Traces tell you where. OpenTelemetry is the vendor-neutral standard for instrumenting distributed systems with traces, and it's particularly useful for job queues because a single job often touches multiple services (the enqueue call, the worker pickup, a downstream API call, a database write). Instrumenting the queue path with OpenTelemetry means you can follow one slow job end to end instead of guessing which hop added the latency.
4. Sentry
Job failures that only show up in a log file get ignored. Sentry catches unhandled exceptions in worker code and groups them by root cause, so a job that's failing for the same underlying reason a hundred times a day shows up as one issue, not a hundred lines of noise. The free tier is generous enough for most background job workloads, and the alerting integrates with Slack and email without extra setup.
5. Flower
If your stack runs on Celery, Flower is purpose-built for exactly this job. It's a lightweight web UI that shows active workers, task history, queue depth, and lets you inspect (or even revoke) individual in-flight tasks in real time. It's not a full observability platform, but for a quick "what is my queue actually doing right now" view, it's faster to stand up than anything else on this list.
"Teams instrument the API and skip the queue every time, and the queue is usually where the actual user-facing delay is hiding." - Dennis Traina, founder of 137Foundry
How these five tools fit together, end to end
It helps to think of these as a pipeline rather than five independent options to choose between. Prometheus is the collection layer: it's what actually scrapes and stores the raw numbers, queue depth, job duration, retry count, at a fixed interval, and everything else in this list either feeds it or reads from it. Grafana sits on top as the visualization layer, turning those stored time series into the dashboard someone actually looks at during a standup or an incident. OpenTelemetry operates alongside both, at the trace level rather than the metrics level, answering "where did the time go within this one job" rather than "how is the queue trending over the last hour." Sentry and Flower are more specialized: Sentry catches the unhandled exceptions that never should have made it to a metrics dashboard in the first place, and Flower (if you're on Celery specifically) gives you a live operational view for the "what is happening right now" question that a historical dashboard answers poorly.
None of the five require you to adopt all of them simultaneously. A reasonable starting point for a team with nothing in place today: get Prometheus and Grafana running first, since that pair alone turns "the queue feels slow" into an actual number you can point at. Add Sentry next if unhandled exceptions in worker code are currently only visible in raw logs. OpenTelemetry and Flower (or your queue's equivalent live-inspection tool) are worth adding once the first two are established and you're debugging specific slow jobs rather than just tracking overall health.
Self-hosted versus managed versions
All five of these have a free, self-hosted path, which is what makes them worth listing here, but most also offer a managed or cloud-hosted tier if running your own Prometheus and Grafana instances isn't something your team wants to own. Grafana Cloud and hosted Prometheus-compatible services exist specifically for teams that want the dashboards without the operational overhead of running the storage layer themselves. Sentry's hosted tier is, for most teams, the default choice over self-hosting anyway, since self-hosted Sentry is a genuinely heavier piece of infrastructure to maintain than the value it captures for a small queue. Start with whichever version gets you real visibility fastest; you can always migrate between self-hosted and managed later without changing how your jobs are instrumented.
What to actually alert on with these tools
Having the data flowing is only half the job; the other half is deciding what should wake someone up versus what's just useful for a retrospective. Queue depth trending upward for an hour is worth an alert. A single spike that clears in two minutes usually isn't. Retry rate crossing a threshold (say, more than 5% of jobs needing more than one attempt) is a much better early-warning signal than raw failure count, because it catches degradation before jobs actually start failing outright. Most teams get this backwards at first, alerting on absolute counts that scale with traffic instead of rates that stay meaningful as volume grows.
Worker saturation is the other metric worth dashboarding explicitly: how many workers are currently busy versus idle, sampled continuously. A queue that's growing while workers sit idle points at a completely different problem (a stuck job holding a worker, a crashed consumer that isn't restarting) than a queue that's growing because every worker is maxed out and there simply aren't enough of them. The five tools above will surface both signals if you build the dashboards to separate them; the default dashboard templates that ship with most of these tools usually don't split queue-growth-from-idle-workers out as its own panel, so it's worth adding manually.
Wire it up before you need it
None of these tools require a large time investment individually, but the value compounds when they're all pointed at the same queue: Prometheus for the numbers, Grafana for the trend lines, OpenTelemetry for tracing a specific slow job, Sentry for catching the exceptions nobody's watching for, and Flower (or your queue library's equivalent) for a quick real-time check. The alternative is finding out your queue has been backing up for six hours because a customer complained first. We covered the retry and persistence side of building a reliable queue in our guide to a background job queue that survives a server restart; monitoring is the half of the story that tells you when that reliability is actually being tested.
More on how we think about production-grade backend infrastructure at 137foundry.com.
Top comments (0)