A client's job queue kept spiking to a few thousand pending items every afternoon, and the autoscaler responded exactly as configured: it added workers. Cost went up, the backlog cleared a little faster, and the same spike showed up again the next afternoon like nothing had changed. Queue depth was the wrong number to scale on, and it took us longer than it should have to admit that.
Photo by Eric Stoynov on Unsplash
What queue depth actually measures
Queue depth tells you how many jobs are waiting. It says nothing about how long they'll take, how urgent they are, or whether the backlog is growing or shrinking. A thousand jobs that each take fifty milliseconds is a completely different situation from a hundred jobs that each take two minutes, and a raw count treats them identically. Scaling decisions made on that single number end up reacting to the wrong thing more often than they react to the right one.
In the client's case, the afternoon spike was a batch of low-priority report jobs landing all at once from a scheduled export, not an increase in real user demand. The autoscaler saw a big number and added capacity for work that had no deadline at all, while the actual time-sensitive jobs, a much smaller and steadier stream, never needed the extra workers in the first place.
The metric that actually mattered
What we switched to was oldest-job age: how long has the single oldest pending job been waiting, broken out per job type rather than as one blended number across the whole queue. That number answers the question an operator actually cares about, is anything waiting too long, directly, instead of forcing someone to infer it from a count that doesn't distinguish urgent work from bulk work.
def oldest_pending_age(queue, job_type):
jobs = queue.pending(job_type=job_type)
if not jobs:
return 0
oldest = min(jobs, key=lambda j: j.enqueued_at)
return now() - oldest.enqueued_at
Once autoscaling read from this number per job type instead of total queue depth, the afternoon batch stopped triggering scale-up events entirely, because those report jobs had no age threshold they were violating. Nobody was waiting on them.
Why this took a while to notice
The reason queue depth is the default metric almost everywhere is that it's the easiest number to expose. Most queue backends, whether that's Redis lists or a database table, return a count with a single cheap query, while oldest-job age per type requires either an indexed timestamp column or a bit more work against the broker's API. The easy metric became the default metric, and the default metric became the thing dashboards and autoscaling rules were built around before anyone stopped to ask whether it actually correlated with the problem it was meant to catch.
"Every autoscaling rule is really a bet about what a specific number implies. The number one client's dashboard treated as 'the queue is struggling' turned out to mean nothing more than 'a scheduled job ran,' and nobody had questioned that assumption in over a year." - Dennis Traina, founder of 137Foundry
Checking the assumption before trusting the new metric
Before committing to oldest-job age as the replacement signal, we spent a couple of days validating that it actually correlated with the thing we cared about, real user-facing delay, rather than assuming a more sophisticated-sounding metric was automatically better than the simple one. We pulled a sample of incidents from the previous quarter where users had genuinely complained about slowness and checked what oldest-job age looked like at those moments versus what raw queue depth looked like. Age tracked the complaints closely. Depth didn't track them at all, spiking just as often on quiet, complaint-free days as it did during actual incidents.
That validation step mattered more than it might sound like it should. It's easy to swap one metric for another that sounds more precise without confirming it actually measures the right thing, and a wrong assumption baked into an autoscaling rule is expensive to unwind later, once dashboards and alerting are already built around it.
What we changed in practice
The fix wasn't a rewrite of the autoscaler, just a change to what it reads. We added a per-job-type oldest-pending-age gauge, set a threshold specific to each job type's actual latency requirement (thirty seconds for payment confirmations, twenty minutes for batch reports), and pointed the scale-up trigger at that instead of the blended count. Total worker count during a normal day barely changed. What changed was that scale-up events started correlating with genuine user-facing delay instead of firing on bulk work nobody was waiting for.
We also kept a secondary, much looser rule on raw depth as a safety net, because an unbounded queue eventually means something regardless of age, even if every individual job is technically still within its threshold. But it's a backstop now, not the primary signal.
Where this connects to the durability side
Autoscaling and job durability are separate problems that tend to surface around the same maturity point in a job system's life. We wrote up the durability half, what actually keeps a job from vanishing if a worker process restarts mid-task, in a longer guide on building a background job queue that survives a server restart. Getting the metrics right doesn't help if the underlying queue can silently lose work; both pieces need to be solid before the system earns any trust from the team that has to operate it.
What we watch for after making this kind of change
Switching the scaling signal isn't a one-time fix you can walk away from. We keep a close eye on two things for a few weeks after: whether the new per-job-type thresholds still hold as traffic patterns shift, since a threshold tuned for today's volume can become wrong within a quarter as a product grows, and whether any job type was left off the per-type breakdown entirely, which quietly falls back to whatever default behavior existed before the change. Cloud providers' own autoscaling documentation, including AWS's guidance on Auto Scaling and Google Cloud's autoscaling documentation, both make the same point in different words: a scaling policy is only as good as the signal it's built on, and revisiting that signal periodically matters more than getting the initial threshold perfect.
A secondary check we added after the fact
A few weeks after switching the primary signal, we added one more check almost as an afterthought: comparing the number of active workers against the number workers actually being utilized, versus sitting idle waiting on a lock or a downstream dependency. It turned out a meaningful chunk of the "capacity" the old autoscaler had added during those afternoon spikes was never doing useful work in the first place, workers were up and billed for, but blocked waiting on a database connection pool that hadn't scaled alongside them. That's a distinct problem from the metric itself, but it's exactly the kind of thing that only becomes visible once you stop trusting the first number that looks like it explains the spike.
The takeaway
If your autoscaling rule reads a single blended queue-depth number, it's worth checking what's actually driving the spikes before adding a scaling threshold on top of it. A number that goes up because a scheduled batch ran and a number that goes up because users are waiting look identical in a total count and require completely different responses. Splitting metrics by job type and switching to age instead of volume is usually a smaller change than it sounds, and it's the kind of instrumentation work 137Foundry's automation team ends up doing on most client job systems eventually, once the blended metric stops telling the operator anything useful.
Top comments (0)