Asked in an LLM inference systems engineer interview.
What they are really testing:
Whether you think about inference as a system with measurable levers (throughput, latency, quality, GPU utilisation) rather than a list of tricks, and whether you can tie each optimisation to the SLO it protects.
A strong answer:
- Start by fixing the targets: TTFT and inter-token latency SLOs per tier, quality bar, and cost per million tokens. Without these, every optimisation is a guess.
- Serving efficiency first: continuous batching, paged KV cache (vLLM-style) to cut memory fragmentation, and prefix caching for shared system prompts. These raise tokens per GPU-second without touching model quality.
- Model-level levers: quantisation (FP8 or INT8 weights, INT4 where quality holds), speculative decoding for latency-bound decode, and distillation or smaller models for easy traffic.
- Routing: a classifier or cascade that sends simple requests to a small model and escalates on low confidence. Measure the escalation rate and the quality delta on an eval set, not just average cost.
- Capacity and scheduling: right-size tensor parallelism to the model and hardware, separate prefill and decode pools when prompts are long, use spot or preemptible capacity for batch and async workloads, and autoscale on queue depth rather than CPU.
- Controls: cap max output tokens, enforce prompt length budgets, cache deterministic responses where semantically safe, and track cost per request by feature so regressions show up in dashboards.
Common mistakes:
- Quoting a percentage saving for quantisation or caching without checking it against your own eval suite and latency targets.
- Optimising GPU utilisation alone, which can raise p99 latency and break user-facing SLOs.
How would you order these levers for a team that is already in production with a fixed latency budget? Share your approach in the comments.
Originally shared on LinkedIn.
An AI interview question every day, leadership for tech people and hands-on AI learning:
Top comments (0)