System Design Interview: Your Queue Is Growing. Will More Workers Actually Help?

A small backlog calculation that makes scaling, downstream limits, and recovery time much easier to explain.

You add a queue to the architecture, draw a few workers behind it, and feel the design becoming pleasantly sensible. Requests no longer wait for every background task, and a traffic spike has somewhere to go.

Then someone asks what happens when jobs arrive faster than the workers can finish them.

Adding workers sounds like a perfectly reasonable answer. Sometimes it is exactly the right answer. But if every worker is waiting for the same overloaded database, you may have drawn a larger crowd around the same small door.

I prefer doing a little arithmetic before adding another box to the diagram. It tells us whether the proposed fix can actually close the gap, which is rather helpful when the gap is growing every second.

Let’s use an original practice scenario with simple numbers. No company interview claim, no imaginary production incident, and no queueing-theory exam hiding around the corner.

Give the queue a job and the user a promise

Imagine a service that accepts background jobs to generate customer reports. Once accepted, a job must remain recoverable until it succeeds or reaches an explicit failed state that the user can see.

Acceptance does not mean the report is ready. The interface returns a job ID, and the user checks its status or receives a completion notification. We also need to agree how long waiting is acceptable, because a successfully queued report can still arrive too late to be useful.

For the arithmetic, assume equal-cost jobs, steady rates, no retries, no duplicates, and no startup delay. Count unfinished work across the queue and workers, rather than pretending that fetching a message means completing it. These assumptions make the example easy to check; real measurements will be messier.

Our incoming rate is 1,200 jobs per second. We have 40 workers, and each can complete 20 jobs per second when its dependencies are healthy. The database can support at most 1,000 completed jobs per second for this workload.

That last number is part of this hypothetical scenario, not a general database limit.

Find the gap before choosing the fix

Forty workers completing 20 jobs each gives us 800 completions per second. With 1,200 new jobs arriving, unfinished work grows by 400 jobs every second.

After five minutes, the additional backlog is 400 multiplied by 300, or 120,000 unfinished jobs.

The queue has done something useful by holding the work. It has not created the capacity to finish it.

This is why a healthy enqueue endpoint is not enough evidence that the overall service is healthy. Our API can keep accepting jobs quickly while the completion promise gets further away.

For this simple model, write the relationship as backlog growth per second equals arrivals minus successful completions, while a backlog exists. Keep every rate in the same units and count real completed jobs, not polling calls or processing attempts.

Already, the discussion has become more specific. We need to close a 400 job per-second gap before we can even begin catching up.

Double the workers, then check what they share

Now double the fleet to 80 workers. Their standalone potential is 1,600 jobs per second, which looks comfortably above the arrival rate.

Unfortunately, every completed report still needs the database, and our assumed database ceiling remains 1,000 jobs per second. Under this simplified model, effective capacity is limited by the smaller of worker capacity and downstream capacity.

So scaling helps us move from 800 to 1,000 completions per second. That is an improvement, but it still leaves a 200-job-per-second deficit against 1,200 arrivals. The backlog grows more slowly; it does not drain.

In a real service, pushing past a dependency’s useful concurrency can also increase contention or timeouts. The neat ceiling in our example is generous enough to keep that complication out of the calculation.

Before scaling, inspect where workers spend time. Are they doing CPU work, waiting for database connections, hitting an external rate limit, or sitting behind a hot partition? More workers are a proposal about the limiting resource, so the answer needs evidence about that resource.

If the database is the limit, possible changes include reducing the work each job asks it to do, changing the access pattern, or increasing its usable capacity. Each needs its own correctness and performance check. A larger worker fleet cannot substitute for that investigation.

Catching up is a different calculation from keeping up

Suppose we start recovery with the 120,000 unfinished jobs calculated earlier. Traffic now falls to 700 jobs per second, and the scaled system can sustain 1,000 successful completions per second.

It is tempting to divide 120,000 by 1,000 and declare that we will recover in two minutes. That calculation quietly assumes new work has stopped arriving.

But 700 of those 1,000 completions are needed just to keep up with new arrivals. Only 300 per second are available to reduce the backlog.

Our estimated catch-up time is therefore 120,000 divided by 300, or 400 seconds. That is six minutes and forty seconds in the constant-rate model.

If arrivals stayed at 1,000 per second, there would be no spare capacity to clear the accumulated work. If they remained above 1,000, the backlog would continue growing.

This estimate describes when the total backlog clears. It is not a prediction of an individual report’s waiting time or a p99 latency promise. Job sizes, scheduling, bursts, and startup delays all affect what particular users experience.

Try changing one assumption before moving on. If new arrivals fall to 400 per second, spare capacity becomes 600 and the same backlog clears in 200 seconds. I like this follow-up because it checks whether the reasoning survived after the first set of numbers disappeared.

Decide what happens before the backlog becomes a broken promise

Once we know the gap, we need a policy for work the system cannot finish on time. That decision belongs beside the capacity calculation, not after an optimistic autoscaling arrow.

For our report service, a reasonable starting proposal is to limit admission when the estimated delay exceeds the product’s agreed budget. A rejected request must receive a clear response rather than an accepted job ID that will never be honored. Already accepted reports still need a recoverable outcome.

Some products prefer to keep accepting work and tolerate longer delays. Others allow obsolete work to be cancelled or coalesced. Those choices depend on the contract; throwing away accepted work merely because a graph looks uncomfortable is not a recovery strategy.

Amazon’s discussion of queue backlogs describes the recovery risk of accumulated work and the importance of measuring message age. It is useful context for why a queue’s durability does not remove its latency trade-off.

For the report example, I want a small operational picture that answers concrete questions.

• Track arrivals and successful completions in jobs per second so the widening or narrowing gap is visible.
• Track the age of the oldest waiting job and end-to-end completion latency so waiting users do not disappear behind a depth chart.
• Track queued, in-flight, and failed work separately so moving messages around does not look like finishing them.
• Track dependency latency and saturation alongside worker utilization so scaling follows the actual constraint.

A queue of 1,000 tiny reports and a queue of 1,000 huge reports are not equivalent. Count is useful, but the amount of work and the age of that work matter too.

Bring retries back into the picture carefully

We excluded retries from the arithmetic because they deserve their own line in the answer. A failed attempt still uses resources, and a retry can use them again without adding a new user job.

Suppose the database starts timing out and workers immediately retry. Attempt traffic can rise even while successful completion throughput falls. Looking only at how busy the workers are would be terribly flattering to the wrong metric.

Bound retry attempts or elapsed retry time, apply suitable backoff, and keep failures visible. If a job is moved to a dead-letter queue for investigation, it has not become a successfully generated report. Define who owns recovery and what the user sees.

Also separate throughput from correctness. If delivery can happen more than once, the report’s externally visible effects still need duplicate protection. Processing the same job twice does not create two units of useful customer progress.

Practise the explanation without hiding behind the diagram

Take one background-processing problem from this interview question collection, then add your own explicit arrival rate, worker rate, and dependency ceiling. Keep your assumptions separate from the original prompt.

Explain how fast unfinished work grows, whether scaling changes the limiting resource, and how long recovery takes while new work continues to arrive. Finish by describing what happens to accepted jobs when the waiting-time budget is exceeded.

For our scenario, the answer is concrete. Forty workers finish 800 jobs per second, so five minutes at 1,200 arrivals creates 120,000 unfinished jobs. Doubling workers helps, but the database caps completions at 1,000. When arrivals fall to 700, the backlog takes 400 seconds to clear under our assumptions.

That is a much more defensible design conversation than adding workers and hoping the queue feels better about it.

I still want the queue in the architecture when it serves the contract. I just want the recovery calculation beside it, because storing a promise and keeping it are two different pieces of engineering.

~ Emily