The Inference Engineering Masterclass — Philip Kiely & Ali Taha, Baseten

The same open-weights model, served by different providers, can differ by 4x to 10x in speed. So "which model" only answers half the question. The other half is whose inference.

Baseten's Philip Kiely and Ali Taha went through the whole stack on Latent Space. A few things worth stealing:

What actually happens to a 200k-token request

The first question is not "which GPU." It is "have you sent me this before?" Cache-aware routing looks for a replica with free prefill workers and some of your input already cached, so it can skip part of prefill. Then prefill and decode run on separate GPU sets. A speculator model sits in front, and if it was trained assuming you write code, your draft-token acceptance rate is high. Ask it to summarize every Harry Potter book instead and it slows down.

The counterintuitive one

Industry common sense says quantization is lossy, so compressing harder is strictly worse. Baseten's research shows quantization errors can cancel each other, and you can predict which layers will cancel. Quantize those. A model with layers 1, 5 and 10 quantized can beat one with only layers 1 and 2. Their GLM-5.2 quant ends up 20 percent more quantized than NVIDIA's, which means 20 percent more throughput at better quality. They proved it with KL divergence against the full-precision logit distribution, not benchmarks.

How the gains stack

A 1T-parameter model on an off-the-shelf engine with no speculator, no KV routing, no disaggregation runs around 30 to 50 tokens per second. Optimized, 300 to 400. The multipliers: BF16 to NVFP4 is roughly 2x, speculative decoding roughly 2x, prefill/decode disaggregation roughly 2x, plus double digits from better kernels. Ten times is aggressive. Four to six is realistic. Hold hardware constant and pure inference work gets you 2x to 4x. Speculation plus quantization is about 95 percent of it.

How to tell when a field is mature

My favorite line was about exactly this. In finance you measure progress in basis points, and five is big news. In inference the numbers are still 20 percent, 100 percent, 200 percent. You will know inference is solved when researchers start publishing about getting 1 percent faster.

Takeaway

For anyone building on model APIs, this is uncomfortable. Models are swappable. Inference quality is not. Which means the real decision was never just which model. It is which model, on whose inference, at what price.

---

Based on the Latent Space episode with Philip Kiely and Ali Taha of Baseten. Technical details per the original episode and Baseten's published research.