What Is LLM Inference?

Inference is what happens every time you send a prompt to an AI model: the model uses what it already learned to produce an answer, without learning anything new.

You hear the word everywhere. "Inference costs", "inference providers", "fast inference". It sounds technical, but the idea is simple. If training is a student studying for years, inference is that student answering an exam question. The studying is done; now they just use what they know.

This article explains what inference is, how it differs from training, what happens inside one request, and why it has become one of the hardest engineering problems in AI.

Training vs inference

Training creates the model; inference runs it.

A model's knowledge lives in its weights: billions of numbers that encode the patterns it found in its training data. During training, the model reads huge amounts of text and keeps adjusting those weights to get better at predicting the next token (a token is a chunk of text, roughly a word or part of one).

Once training is done and the model is deployed, the weights are frozen. When you ask it "Why do cats purr?", it does not update itself on your question. It runs your input through those fixed weights and produces an answer. That run is inference.

	Training	Inference
Purpose	Learn the weights	Use the weights to answer
Weights	Change constantly	Fixed
When it happens	Once per model version, during development	Every single request, for as long as the model is served
Input	Massive datasets	One prompt (plus chat history, system instructions, documents)
Main cost driver	A huge one-off compute run	Volume: millions of requests a day
What users notice	Nothing directly	Speed, cost per token, availability

A common misconception: chatting with a model does not "teach" it. If an app seems to remember you, that is the application storing your history and sending it back in the prompt, not the model's weights changing.

What happens in one inference request

Every request goes through the same five stages, and only one of them, decode, runs in a loop.

[embedded content: one inference request · 5 stages, decode loops]

Tokenize turns your text into token IDs. Prefill reads all of them in parallel and stores intermediate attention data in the KV cache, so it is not recomputed later. Decode then writes the answer one token per pass through the model, reusing that cache each time. Streaming sends each token to your screen as soon as it exists.

Prefill is fast because the whole prompt is already known. Decode is slow because token 10 cannot be chosen until token 9 exists. That single difference shapes almost every design decision in inference systems.

Why inference is the hard part at scale

Training is paid once; inference is paid on every request, forever, so at scale it becomes the bigger engineering problem.

Serving one user is easy. Serving millions at once raises three problems:

1. Speed. Each new token needs a full pass through the model, and each pass reads the model's weights from GPU memory. For large models, moving that data, not the maths, is often the bottleneck.
2. Memory. Every active conversation keeps its own KV cache (stored attention data for each token so far) in GPU memory. Longer chats and more users mean more cache, and GPU memory runs out fast.
3. Cost. GPUs are expensive. An idle GPU wastes money; an overloaded one makes users wait.

Inference systems attack these with a few core techniques:

• Batching: run many users' requests through the model together, so one read of the weights serves many people.
• KV cache management: allocate cache memory in small blocks instead of reserving one huge chunk per request, so more requests fit on one GPU.
• Quantization: store weights in fewer bits (for example 8-bit instead of 16-bit) to cut memory use and speed up reads, at a small accuracy cost.
• Distributed serving: split a model too large for one GPU across several.

Every one of these is a tradeoff between how fast one user gets an answer and how many users one GPU can serve.

How inference speed is measured

Three numbers describe how an inference system performs, and they matter to different people.

Metric	What it measures	Who cares most
Time to First Token (TTFT)	Time from sending a request to receiving the first token. Includes network, queueing, and prefill.	Users of chat apps: it decides whether the app feels responsive
Tokens per second (per request)	How fast one answer is written once it starts. Driven by decode.	Users reading long answers, agents chaining many calls
Throughput	Total tokens the system produces per second across all users	The team paying for the GPUs

These pull against each other. Packing more requests into one batch raises throughput (cheaper per token) but can make each individual request slower. A good inference setup picks the balance that fits the product: a chat app favours low TTFT, an overnight batch job favours throughput.

Wrapping up

Inference is the model at work: frozen weights, your prompt in, tokens out, one step at a time.

Training gets the headlines, but inference is what you touch every time you use AI. It decides how fast answers appear, how much each token costs, and how many people a provider can serve at once. Most of the engineering behind modern AI products, from streaming to batching to KV cache tricks, exists to make this one loop faster and cheaper.

In the previous article, we looked at why models stream their answers. That was one piece of the inference story: making the slow decode loop feel fast.