You type a prompt into ChatGPT. Two seconds later, a full answer appears. Nothing about the model changed during those two seconds — it didn't learn anything, it didn't get smarter, it didn't update itself based on your question. So what actually happened in that gap?
That's inference. It's the least talked-about part of AI, and it's also the part you interact with every single time you use an AI product. Training gets all the headlines — the huge datasets, the massive GPU clusters, the eye-watering costs. Inference is the quiet workhorse that runs millions of times a day, for every user, every request, forever.
In this post, you'll learn what inference actually is, how it's different from training, what happens under the hood when a model generates a response, and the misconceptions that trip up almost everyone when they first hear the term.
The Restaurant Kitchen Analogy
Imagine a chef going through culinary school. For years, they practice recipes, taste thousands of dishes, adjust seasoning, throw out what doesn't work, and slowly develop a set of recipes they trust. This process is slow, expensive, and happens in a controlled environment — a training kitchen, not a real restaurant. Nobody is being served yet.
That's training. It's the phase where a model looks at huge amounts of data, adjusts its internal "recipe" (its weights, the numbers that define how it behaves), and gradually gets better at a task.
Now the chef opens a restaurant. A customer orders the dish. The chef doesn't re-invent the recipe or run new experiments — they use the recipe they already perfected, and they cook it, fast, consistently, correctly, over and over, for every customer who walks in. No more learning happens during dinner service. The recipe book is closed and finalized.
That's inference. It's the model using what it already learned to produce an answer for you, right now, without changing anything about itself in the process.
How It Actually Works
Once training finishes, a model's weights are frozen — the recipe book is locked. Inference is simply feeding new input through that frozen model and reading out what comes on the other end. Engineers call this a forward pass: your input goes in one end, gets transformed step by step through the model's layers, and a result comes out the other end. There's no backward step, no correcting mistakes, no updating the recipe. Just: input in, output out.
For a chatbot, there's an extra wrinkle worth knowing. Large language models don't generate a whole response in one shot — they generate it one token at a time (a token is roughly a word or a word-fragment). It's like the chef plating a dish one element at a time: place the protein, step back, look at what's on the plate so far, decide what goes next, add the sauce, step back again, decide again, and keep going until the dish looks done. Each new token the model produces gets fed back in as part of the input for producing the next one. That's why longer answers take visibly longer to appear — the model is doing this "look at everything so far, decide the next piece" step again and again.
A Concrete Example
Say you ask a model: "Write a haiku about the ocean."
First, your sentence gets chopped into tokens — this is like a prep cook breaking ingredients down into usable pieces before cooking starts. Those tokens get converted into numbers the model can process, and pushed through the frozen network in a forward pass. The output isn't a word — it's a probability distribution over every possible next token, essentially the model saying "here's how likely each possible next word is, given everything so far." The system picks one (usually the most likely, sometimes with a bit of controlled randomness), appends it to the growing response, and repeats the whole process to pick the next token. This continues until the model produces an end-of-response signal or hits a length limit.
Every one of those steps costs compute time — this is where terms like latency (time to get an answer), throughput (how many requests a system can serve per second), and cost per token come from. Techniques you'll hear about in production AI — quantization (using smaller, less precise numbers to run faster, like using a simplified prep technique that's a little less exact but much quicker), batching (cooking several orders together instead of one at a time), and caching (keeping frequently-used prep work ready instead of redoing it from scratch) — all exist specifically to make inference cheaper and faster. None of them touch training at all.
Common Misconceptions
"Using a model teaches it more." Not by default. A production model doesn't update its weights based on your conversation. The recipe book stays closed. (A company might later use logged conversations to train a future version — but that's a separate training run, not something happening live while you chat.)
"A bigger model is always the better choice." Bigger models often produce better answers, but they're also slower and more expensive to run at inference time. In production, teams frequently choose a smaller or compressed model on purpose, because a slightly-less-brilliant answer in 200 milliseconds beats a slightly-better one in four seconds.
"Once training is done, the hard part is over." For any AI product with real users, inference — not training — is usually where the bulk of ongoing computing cost goes. Training happens occasionally. Inference happens every single time someone hits "send."
What to Try Next
If you want to actually feel the difference between model sizes at inference time, install Ollama or LM Studio and run a small open model locally (something like a 1–3 billion parameter model). Watch the tokens stream in one at a time, then try a larger model on the same machine and notice how much slower it feels. That speed difference, multiplied across millions of users, is the entire reason inference optimization is its own engineering discipline.
Training teaches a model what to know. Inference is everything that happens after — every time it actually gets used. Once you can see that difference, a lot of how AI products are actually built (and why they're priced the way they are) starts making a lot more sense.
If this helped you understand inference, drop a reaction.
Top comments (0)