DEV Community

Manohar Papasani
Manohar Papasani

Posted on

What optimizations would you employ to achieve high throughput and low latency?

#aiinterview #llminference #genai #aiarchitect

Asked in an LLM inference systems engineer interview.

What they are really testing: whether you separate the two goals (latency per request vs. tokens per second across the fleet), know where LLM serving time actually goes, and can name concrete mechanisms for hiding data movement behind compute.

A strong answer:

  • Split the problem by phase. Prefill is compute-bound and benefits from batching and chunked prefill; decode is memory-bandwidth-bound and is dominated by reading weights and KV cache. Optimize each with different levers.
  • Use continuous (in-flight) batching so finished sequences leave the batch and new ones join at every decode step, instead of waiting for the slowest request in a static batch.
  • Manage KV cache with paged allocation (PagedAttention-style blocks) to cut fragmentation, and reuse shared prefixes with prefix caching. Offload cold KV blocks to host memory or NVMe only when the recall cost is below recomputation cost.
  • Quantize weights and KV cache (FP8 or INT8, INT4 for weights where quality holds) to shrink memory traffic, and use FlashAttention-style fused kernels to cut HBM round trips.
  • Overlap transfers with compute: pin host memory, issue host-to-device copies on a separate CUDA stream with events for synchronization, prefetch the next layer's weights or KV blocks while the current layer runs, and use CUDA graphs to remove launch overhead.
  • Use speculative decoding or multi-token prediction when the draft acceptance rate is high enough to pay for the draft model's cost, and consider tensor or pipeline parallelism only when a single model does not fit or when latency targets demand it, since communication adds its own overhead.

Common mistakes:

  • Optimizing only average throughput and ignoring time-to-first-token and inter-token latency percentiles under real load.
  • Offloading KV cache to CPU without measuring PCIe bandwidth and transfer latency, which can make decode slower than recomputing.
  • Increasing batch size indefinitely, which raises queuing delay and memory pressure and can hurt tail latency.

How would you answer this? I'm curious which trade-off you would tackle first in a production setting.

Originally shared on LinkedIn.

An AI interview question every day, leadership for tech people and hands-on AI learning:

Top comments (0)