https://github.com/Flow-Research/local-first-AI/pull/13#issuecomment-5085695479
I've recently begun reading the book Inference Engineering by Philip Kiely together with the interactive tutorial: inferenceengineering.tech (https://inferenceengineering.tech/). Chapters 0 and 1 are short, but they had an impact on my thinking when it comes to “making AI faster.” Prior to this I was more or less past the point of looking at engines, GPUs and knobs. One of these two chapters had me forcefully shifting gears and taking a step back.
Chapter 0 provides the following definition of inference: The life of a generative model comes in two phases
training (where the weights are learned), and inference (where production is based on those weights). It might be very easy to be a classic ML inference. Generative AI inference is NOT. You can't just download weights and rent a gpu and expect something to be fast, inexpensive and scalable at all.
The core mental model is three layers that have to work together:
| Layer | What it is really about | Rough question it answers |
|---|---|---|
| Runtime | One model on one GPU-backed instance (or a few GPUs in one box) | How do I squeeze maximum performance out of this machine? |
| Infrastructure | Clusters, regions, clouds, autoscaling, capacity | How do I survive traffic that exceeds one box? |
| Tooling | The developer experience and abstractions on top | How much control do engineers get without drowning in primitives? |
That sounds obvious, but it is easy to forget in practice. If a single vLLM process is already efficient and users still wait, the bottleneck may be queueing, cold starts, or multi-region capacity — infrastructure, not another CUDA trick. If the platform is powerful but nobody can operate it safely, tooling is the missing layer.
Chapter 1: Do not optimize before you know what “best” means
Chapter 1 is titled Prerequisites, and the message is blunt: optimization means choosing among tradeoffs. Latency, throughput, quality, and cost pull in different directions. Elite athletes are not the absolute biggest, fastest, and strongest at once; they are specialized. Inference services are the same.
Before dedicated inference work is worth it, you should be able to answer five product questions:
| Question | Why it matters |
|---|---|
| Which model(s)? | Model size and architecture dominate cost and speed |
| What interface? | Chat streaming, tool calls, and batch jobs need different metrics |
| What latency budget? | TTFT for chat is not the same as total time for an agent step |
| What unit economics? | Per-request or per-user spend caps what you can afford |
| What usage patterns? | Concurrency and spikes decide shared API vs dedicated GPUs |
I also liked how the chapter forces use-case specificity. Online chat wants low latency. Offline transcription wants high throughput. Consumer apps care more about marginal cost and elasticity. B2B apps often need predictable latency and uptime. One model can serve both online and offline traffic, but two differently tuned deployments are often cheaper once volume exists.
Metrics finally made sense
Chapter 1’s measurement section connected cleanly to Chapter 0’s prefill/decode story:
| Metric | Meaning | Phase it mostly reflects |
|---|---|---|
| TTFT | Time until the first output token | Prefill |
| TPS / ITL | How fast tokens arrive after the first one | Decode |
| P90 / P99 | Tail latency, not just the average | Real user trust |
| E2E latency | Network + queue + inference | What users actually feel |
Averages lie. A product can feel broken if one request in ten is slow, even when the mean looks fine. And if pure GPU time is good while end-to-end is bad, stop retuning kernels and inspect infrastructure.
What I am taking into the rest of the book
Chapters 0 and 1 did not teach me how to configure vLLM yet. They taught me a better order of operations: define the product constraints, pick the smallest model that is good enough, measure the right latency and cost signals, then decide whether runtime work, infrastructure work, or better tooling is the real lever. Inference engineering is not “make everything faster.” It is “make this workload good enough under real constraints.”
I am continuing with Chapter 2 on model architecture and bottlenecks next.
Source: Philip Kiely, Inference Engineering (Baseten Books, 2026); companion site inferenceengineering.tech (https://inferenceengineering.tech/)
Top comments (0)