DEV Community

Manohar Papasani
Manohar Papasani

Posted on

How would you balance this trade-off in practice?

#aiinterview #llminference #genai #mlops

Asked in an LLM inference systems engineer interview.

What they are really testing: whether you can turn a vague throughput-versus-latency tension into concrete scheduler controls, separate traffic classes, and measurable SLOs, rather than optimising one number in isolation.

A strong answer:

  • Start from SLOs per tier: time to first token (TTFT) and inter-token latency for interactive traffic, and completion-time deadlines for bulk traffic. Maximise throughput only within those constraints.
  • Use continuous (iteration-level) batching so new requests join the running batch between decode steps instead of waiting for a full batch to drain. Cap the batch by a token budget and KV-cache memory, not just request count.
  • Use chunked prefill, or split prefill and decode onto separate pools, so a long prompt does not stall decode for every active user and tail latency stays bounded.
  • Separate capacity by class: a low-latency pool with smaller batches for interactive traffic, and a high-batch pool for offline work. Add priority queues and admission control that defers or sheds bulk work first when load spikes.
  • Make batching adaptive: raise the batch token budget when queue depth is high and p95 latency is within target, and lower it when TTFT or inter-token latency breaches the SLO. Sweep batch budgets against load and plot tokens/sec against p95 latency to find the knee.
  • Track goodput, meaning tokens/sec that meet the SLO, rather than raw throughput. Re-run the sweep when the prompt-length mix or model version changes.

Common mistakes:

  • Quoting a throughput figure without the latency percentile it was measured at.
  • Using a fixed static batch size, which wastes GPU capacity under light load or inflates latency during bursts.
  • Putting interactive and bulk requests in one unprioritised queue, so bulk jobs starve or real users see inflated latency.

Which split would you pick first for a mixed chat and batch workload, and which metric would you alert on? Curious how others would answer.

Originally shared on LinkedIn.

An AI interview question every day, leadership for tech people and hands-on AI learning:

Top comments (0)