A worked system design problem about the waiting time your architecture diagram forgot.
Let’s say your model finishes a batch in 100 milliseconds. Someone asks whether the API can respond in 100 milliseconds too, and suddenly the whiteboard feels a little less friendly.
Because the model’s stopwatch starts after the waiting is over. Your user’s stopwatch does not.
Imagine a shuttle promising a ten-minute ride. Lovely. You can still spend forty minutes at the stop while the driver waits for more passengers. Faster driving will not rescue that afternoon.
This is why I like this reported Anthropic interview question: design an LLM inference service where every batch takes 100 ms. The full prompt is readable without an unlock. We will build our own answer, starting with one request rather than a heroic collection of infrastructure boxes.
By the end, you should be able to explain when a batch leaves, how many workers you need, and when accepting another request becomes a promise you cannot keep.
First, find out when the stopwatch starts
Before choosing a queue, clarify what those 100 ms measure. For this walkthrough, assume they cover the model’s complete execution of one batch, not time to the first token and not the client’s entire round trip.
We will also assume one model version, a maximum of 32 requests per batch, one batch at a time per worker, and whole responses rather than streaming. The fixed compute time holds for every supported batch size in this simplified exercise. Those are assumptions for our design, not claims about Anthropic’s production service.
Now give the caller a provisional 200 ms end-to-end budget. Perhaps 20 ms goes to the inbound network and API work, 100 ms to the model, and 20 ms to returning the result. That leaves 60 ms for waiting.
Those 60 ms must cover both deliberate waiting for a fuller batch and waiting because every worker is busy. They are not two separate allowances we get to spend twice.
These numbers are an illustrative budget. They do not prove a p99 target: real network variation, service overhead and arrival bursts still need measurement. If the interviewer means 100 ms end to end while compute alone takes 100 ms, flag that the budget has no room for anything else.
Give a half-empty batch permission to leave
Waiting until all 32 slots fill sounds efficient until traffic gets quiet. One lonely request can sit there while the GPU is perfectly available. I prefer a rule that cares about both fullness and age.
For a tiny example, shrink the batch limit to four and allow at most 10 ms of intentional batching delay. Start with an idle worker:
Request A arrives at 0 ms. Start its waiting clock.
Request B arrives at 7 ms. Keep the original clock; do not restart it.
At 10 ms, send A and B together even though two seats are empty.
Compute finishes at 110 ms. A spent 110 ms in this part of the service; B spent 103 ms.
Now change the arrivals: four requests appear together at 0 ms. Send the full batch immediately, and it finishes at 100 ms. There is no reason to wait out a timer that has already done its job.
That gives us a useful scheduler rule: when a compatible worker is free, dispatch if the batch is full, the oldest request has reached its batching-delay limit, or waiting longer would threaten a request’s deadline. Otherwise, wait for the next arrival or relevant timer.
The timer limits intentional delay; it does not manufacture a free worker. If the only worker is busy until 100 ms, a request that arrived at 1 ms can wait 99 ms before its own compute begins. A 10 ms batch timer cannot make that request meet our earlier budget.
Keep the oldest arrival timestamp while a worker is busy. When capacity returns, dispatch an overdue partial batch immediately rather than starting a fresh 10 ms timer. This is a small implementation detail with a very visible effect on the person waiting.
NVIDIA’s Triton batching documentation describes configurable batching delays and queue policies for stateless models. It is a useful implementation reference; our 10 ms setting is a teaching choice, not a universal recommendation.
Do the capacity math before ordering more GPUs
Under the fixed-time assumption, one worker completes ten batches each second. With 32 requests in every batch, its ideal throughput is 320 requests per second.
For an illustrative arrival rate of 1,000 requests per second, the arithmetic minimum is four workers: 1,000 divided by 320, rounded up. Three workers can finish only 960 requests per second even in this idealized case, so a sustained 1,000-request arrival rate makes the queue grow.
Four workers provide an ideal capacity of 1,280 requests per second, putting the offered load at about 78% of that capacity. Five provide 1,600, bringing it to 62.5%.
Lovely arithmetic, but please do not turn the last number into a latency guarantee. It assumes full batches, zero overhead and no failures. With an average batch size of 16, the same fixed-time model gives only 160 requests per second per continuously busy worker.
Five workers are a candidate configuration to measure, not the answer handed down by the spreadsheet. Track actual batch sizes, oldest queued-request age, accepted and rejected traffic, and end-to-end latency percentiles. Provision headroom for the arrival pattern you actually have.
Autoscaling should consider queued work and deadline pressure, not just GPU utilization. Keep warm capacity when model loading is slow; a worker that arrives after the caller’s deadline cannot save that request.
Now draw the boxes, including the way back
Our first architecture can stay small: client → API frontend → bounded queue for the model version → scheduler → model worker. Results travel back through the frontend to their original callers.
At admission, validate input size and supported settings, authenticate the caller, and attach a request ID, tenant identity, arrival time, deadline and reply destination. Store the waiting connection or future in a request-ID lookup at the frontend. The scheduler does not need to hold an HTTP connection itself.
Return each result with its request ID and attempt identifier. Batch position is a fragile routing contract once retries, cancellations or filtering can change the membership. Resolve the matching waiting request once, and discard stale attempts or late results.
If several model versions are supported, keep compatible queues and route to workers that already have the appropriate weights. Two requests being next to each other in a queue does not make their models compatible. Put tenant quotas or fair scheduling in front of the shared capacity so one enthusiastic caller cannot occupy the entire waiting room.
Decide what admission actually promises
For a synchronous service, we can explicitly allow a frontend crash to fail the connection and require a client retry. If accepted work must survive that crash, we need durable request state, result storage and a reconnect or polling contract. A durable queue alone does not restore a vanished connection.
Bound the queue by count and memory, and expire requests whose deadlines have passed. Reject excess work with an appropriate overload response rather than silently accepting an ever-growing backlog. Use a shared retry budget and backoff with jitter so clients do not all return at the same instant.
An estimated completion time can help admission decisions, but it is still an estimate. If a request cannot plausibly fit within its remaining budget, say so early. Waiting until the deadline to admit defeat wastes both the caller’s time and your capacity.
One broken request should not get unlimited second chances
Happy-path batching is pleasantly tidy. Failure handling is where I want the explanation to become precise.
If a worker dies mid-batch: identify the unfinished requests and retry only those still within their deadlines and attempt budgets. Use leases or equivalent ownership tracking if work can move between workers. Fence old attempts so a late response cannot complete the same request twice.
If one request repeatedly crashes a batch: validate predictable bad inputs before scheduling. For unexplained failures, isolate the suspected request or split the failing group with a strict retry budget. Never keep replaying the same failing group forever.
If a caller times out or disconnects: remove queued work where possible and propagate cancellation to the execution engine when supported. Already-running GPU work may not stop immediately; prevent its eventual output from reviving an expired request.
Inference may execute twice after an ambiguous failure. Completion deduplication protects the response contract, but it does not magically guarantee exactly-once compute. If a client expects retries to replay the original result, keep a tenant-scoped idempotency record with the request fingerprint and saved result. Regeneration is not necessarily identical.
Then let the 100 ms assumption go
Only after solving the stated problem should we relax it. Real generative requests have different prompt lengths and output lengths; request count alone is a poor description of the work or memory they consume.
Consider token and memory budgets, compatible model settings, and separate handling for interactive and bulk work. A short response should not wait for a much longer response merely because they started together.
With continuous batching, an engine can schedule active sequences across decoding steps and admit new work as other sequences finish. vLLM documents continuous batching among its serving features. This changes the scheduler model; it does not make arbitrary generations finish in 100 ms.
For streaming, measure time to first token, time between tokens and full completion separately. A prompt first token can still be followed by a frustratingly slow answer. Keep the metric attached to the experience you promised.
Here is the practice exercise worth keeping
Close the explanation and redraw the system from memory. Use one worker, a batch limit of four, a 10 ms batching delay and 100 ms compute. Then trace these three situations by hand:
Sparse arrivals: A at 0 ms and B at 7 ms. Explain why the batch leaves at 10 ms and why the callers wait different amounts.
A busy worker: four requests start at 0 ms; E arrives at 1 ms. With no other worker, E can start at 100 ms and finish at 200 ms. Explain why the batch timer did not cap its total wait.
A deadline expires: give E a deadline of 150 ms. Even immediate dispatch when the worker frees up would finish too late. State when you reject or expire it and how you handle any late result.
For these exercises, ignore network and frontend overhead and assume no further arrivals. The point is to make every timestamp defensible before adding more boxes.
My favorite part of this question is that a fast model does not let us skip ordinary engineering. Someone still has to decide who waits, how long they wait, and what happens when the promise no longer fits.
Try the prompt linked near the beginning before reading another solution, and change one constraint yourself.
If these worked interview questions help, follow me or subscribe for the next walkthrough. I would love to hear which policy you choose first whether sending a smaller batch sooner, or wait for better utilization and what would make you change your mind?
~ Emily
Top comments (0)