Most inference optimization content is written for people running fleets of H100s. If you're serving models out of a single GPU box, a modest EC2/ECS setup, or even CPU inference for smaller models, a lot of that advice doesn't transfer — the constraints are different, and the "just add more GPUs" answer isn't available to you.
This post covers the three levers that actually move the needle at small-to-mid scale: KV cache management, quantization, and the latency tradeoffs that come with both. No cluster-scale assumptions.
Why this matters before it's a crisis
Inference cost and latency don't show up as a line item until they do — usually right around the point where a side project or an MVP gets real traffic. At that point you have three options: spend more on hardware, spend more on hosted API calls, or actually understand what's eating your latency and memory budget. The third option is the only one that scales your understanding along with your infrastructure.
Lever 1: KV cache — what it is and why it dominates memory
Every token a transformer generates attends back to all previous tokens. Naively, that means recomputing attention over the whole sequence at every step — wildly wasteful. The KV cache stores the key and value tensors from previous tokens so each new token only needs one forward pass, not a full replay.
The tradeoff: that cache grows linearly with sequence length, and its memory footprint is often the actual bottleneck — not the model weights themselves — once you're serving multiple concurrent requests with long contexts.
A rough sizing formula, per request:
kv_cache_bytes = 2 * num_layers * num_heads * head_dim * seq_len * bytes_per_element
The 2 is for both K and V. For a 7B-class model with 32 layers, 32 heads, head_dim 128, at fp16 (2 bytes) and a 4096-token context, that's already several hundred MB — per concurrent request. Multiply by your target concurrency and you'll see why KV cache, not weights, is usually what determines how many simultaneous users a single GPU can serve.
Practical levers on KV cache size
- Multi-query / grouped-query attention (GQA) — if you're choosing a model, prefer architectures using GQA (most modern open-weight models do). It shares K/V heads across multiple query heads, cutting cache size significantly with minimal quality loss.
- Sliding window attention — cap how far back the cache needs to reach for local-context tasks. Not appropriate for tasks needing full long-range recall, but a real win for chat-style workloads with short working context.
- Cache eviction policies — for multi-turn chat apps, evict or summarize older turns instead of keeping the full cache alive indefinitely. This is an application-level decision, not just an inference-engine setting.
- PagedAttention-style memory management (used in vLLM and similar serving frameworks) — treats the KV cache like virtual memory pages instead of requiring contiguous allocation, which dramatically reduces fragmentation waste when you have many concurrent requests with different sequence lengths.
If you're rolling your own serving layer rather than using vLLM/TGI, this is the single highest-leverage thing to borrow from those projects even if you don't adopt them wholesale.
Lever 2: Quantization — trading precision for throughput
Quantization reduces the numeric precision of model weights (and sometimes activations), shrinking both memory footprint and, often, latency.
| Precision | Relative size | Typical use case |
|---|---|---|
| fp16/bf16 | 1x (baseline) | Default for most hosted APIs |
| int8 | ~0.5x | Good quality retention, solid speedup |
| int4 (GPTQ/AWQ) | ~0.25x | Runs 7B-class models on consumer GPUs |
| GGUF quantized (Q4_K_M etc.) | ~0.25–0.35x | CPU/edge inference via llama.cpp |
The practical decision isn't "should I quantize" — for anything outside a hyperscaler budget, you almost certainly should. It's which quantization scheme, and that depends on your bottleneck:
- Memory-bound (model barely fits, or you want more concurrent requests) → int4 or GGUF gets you the most headroom
- Latency-bound on CPU → GGUF via llama.cpp with an appropriate quant level (Q4_K_M is a reasonable default starting point)
- Quality-sensitive (structured output, code generation, anything requiring precise instruction-following) → stick to int8 or test carefully before dropping to int4; quality degradation isn't linear and some tasks are far more sensitive than others
A pattern worth adopting: don't guess — benchmark your actual task against 2–3 quantization levels before committing. A model that quantizes cleanly on general chat can degrade noticeably on structured extraction or tool-calling tasks.
// Rough shape of a quant-level config in a gateway/router setup
#[derive(Debug, Clone)]
struct ModelVariant {
name: String,
quant: QuantLevel, // Fp16, Int8, Int4Awq, GgufQ4KM
vram_gb: f32,
avg_tokens_per_sec: f32,
}
fn select_variant(task: TaskProfile, variants: &[ModelVariant]) -> &ModelVariant {
// route structured/tool-calling tasks to higher precision,
// route casual chat/summarization to the cheapest variant that fits
variants.iter()
.filter(|v| task.quality_floor <= v.quant.quality_score())
.min_by(|a, b| a.vram_gb.partial_cmp(&b.vram_gb).unwrap())
.unwrap_or(&variants[0])
}
This is the same instinct as multi-provider routing in a gateway — route by task requirements, not a single fixed model for everything.
Lever 3: Latency tradeoffs — where the milliseconds actually go
Once cache and quantization are handled, the remaining latency budget splits into pieces that respond to different fixes:
- Time to first token (TTFT) — dominated by prompt processing (prefill). Long system prompts or heavy RAG context directly inflate this. If TTFT matters more than total generation time for your use case (e.g. interactive chat), trimming prompt bloat is often a bigger win than any inference engine tuning.
- Inter-token latency — dominated by memory bandwidth during decode, which is exactly where quantization and KV cache size have direct leverage.
- Batching — continuous batching (as opposed to static batching) lets new requests join a batch mid-flight instead of waiting for the whole batch to finish, which matters a lot under real, uneven traffic patterns. This is one of the core reasons frameworks like vLLM outperform naive serving loops even before touching quantization.
A concrete tradeoff worth naming explicitly: batching improves throughput but can hurt per-request latency under load, since a request may wait briefly for a batch slot. For a single interactive assistant, that's rarely visible. For a shared production endpoint under real concurrency, it's the difference between p50 and p95 latency looking very different — and it's worth deciding up front which one your use case actually cares about.
Putting it together: a practical decision order
For a small-to-mid scale deployment, the order that gets you the most benefit per hour invested:
- Pick a GQA-architecture model if you have a choice — this is free and structural
- Quantize and benchmark against your actual task, not a generic leaderboard — int8 as a safe default, int4/GGUF if memory-bound
- Adopt continuous batching via an existing serving framework (vLLM, TGI) rather than writing your own naive batching loop
- Trim prompt bloat if TTFT matters for your UX — this is often the cheapest win and the most overlooked
- Only then consider hardware changes — by this point you'll actually know whether you're memory-bound or compute-bound, which determines what upgrade (if any) is worth the cost
None of this requires hyperscaler infrastructure. It requires knowing which of these five levers you're actually constrained by before spending money on the wrong fix.
What's next
The next post in this series moves back up the stack — open-weight versus closed API models, and a practical cost/control framework for deciding which to run yourself versus call through a provider.
This is part of a series on AI infrastructure patterns, drawing on production work building multi-provider AI gateways in Rust. Follow along for the rest of the series.
Top comments (0)