What Is an LLM? A Working Model for Engineers

Originally published on DevToolHub.

An LLM is a model that has one job: given the text so far, predict the next token, append it, and repeat. That's it. Everything else — chat interfaces, coding assistants, agents that call tools — is scaffolding built around that one loop running thousands of times per response.

Most explanations either stop at "it's trained on the internet" or jump straight into transformer math you'll never touch. Neither one helps when you're deciding how much VRAM to buy, why a long conversation suddenly gets dumber, or why your token bill doubled overnight. This is the model you actually need.

What is an LLM?

A large language model is a neural network trained to predict the next token in a sequence, repeated over enough text that it picks up grammar, facts, and reasoning patterns as a side effect of getting good at that one task. Hugging Face's own generation docs put it plainly: an LLM "is trained to generate the next word (token) given some initial text (prompt) along with its own generated outputs up to a predefined length or when it reaches an end-of-sequence (EOS) token."

That's the whole mechanism. There's no separate "understanding" step. The model has learned, from its training data, which token is statistically likely to come next given everything before it — and at large enough scale, "statistically likely" starts looking a lot like reasoning.

Size is measured in parameters — the numeric weights the network adjusts during training. A 7B model has about 7 billion of them. More parameters generally means better output, but it also means more memory to hold the weights and more compute per token generated. That trade-off is the entire reason quantized, smaller models exist.

How does an LLM generate text?

One token at a time, in a loop, with no ability to revise what it already wrote. The model looks at every token so far — your prompt plus whatever it has generated in this response. Then it outputs a probability distribution over its entire vocabulary for what comes next.

Then a decoding strategy picks the actual token from that distribution:

• Greedy search — always pick the single most likely next token. This is the library's default decoding strategy. Deterministic, and prone to repetitive output.
• Sampling — pick randomly from the distribution, weighted by probability, controlled by a temperature parameter. Low temperature (under 0.4) stays close to greedy; high temperature (above 0.8) gets more varied and more prone to nonsense.

That picked token gets appended to the sequence, and the whole thing runs again to pick the next one. This continues until the model emits an end-of-sequence token or hits a length limit you set. There is no lookahead and no backtracking — once a token is chosen, it's part of the context for every token that follows, mistakes included.

This is also why LLMs hallucinate confidently instead of saying "I don't know." The model isn't checking facts; it's continuing a pattern. If the highest-probability continuation of a plausible-sounding sentence is a wrong fact, the model produces the wrong fact with the same fluency as a right one.

What is a token, and why does the context window matter?

A token is the model's actual unit of work — usually a word, part of a word, or a punctuation mark, not a full sentence. "Deployment" might be one token; "unconfigured" often splits into two or three. Everything you feed the model and everything it generates gets converted to and from tokens.

The context window is the maximum number of tokens the model can hold in memory at once — your system prompt, the conversation history, and the response it's generating, all counted together. Run out of room and the oldest tokens get dropped or the request fails outright, depending on how the client handles it.

This is a hard ceiling, not a soft one. A model with a 4,096-token context window doesn't get slower as you approach the limit — it silently loses the beginning of the conversation, or errors out, depending on the client.

Context windows vary by model and by how you run it — Ollama's own docs aren't fully consistent on the default either. Its FAQ quotes a flat "4096 tokens," but the newer context-length docs say the real default scales with GPU VRAM: 4k below 24 GiB, 32k from 24–48 GiB, 256k at 48 GiB and up. Check num_ctx or ollama show <model> rather than assuming — see DevToolHub's Ollama hardware requirements guide for how much RAM and VRAM that adds per model size.

How to measure your LLM's token throughput yourself

Run any local model with the verbose flag:

ollama run llama3.1 --verbose "Explain quicksort in one paragraph"

The output includes a full timing breakdown, including prompt eval rate and eval rate in tokens per second. eval rate is the number that actually determines whether a chat feels instant or sluggish. Ollama's API returns the same data as raw fields on every call — prompt_eval_count, eval_count, prompt_eval_duration, eval_duration, all in nanoseconds — so you can calculate tokens per second yourself with eval_count / eval_duration * 10^9.

Why model size and quantization determine your hardware

A 70B model outperforms a 7B model on complex reasoning, but needs roughly 8x the memory and runs far fewer tokens per second on the same GPU. For narrow tasks — classification, extraction, simple chat — a well-chosen 7B or 12B model often gets you most of the quality at a fraction of the cost.

Quantization shrinks memory footprint by reducing weight precision — from 16-bit down to 4-bit, typically — cutting model size roughly 4x with a modest quality loss. That's why an 8B model tagged Q4_K_M downloads at around 5 GB instead of the 16+ GB its full-precision weights would take.

Common mistakes engineers make with LLMs

• Treating a bigger model as always the right call. For narrow tasks, a smaller tuned model often matches a frontier model's accuracy at a fraction of the latency and cost.
• Not budgeting for context growth. A tool-calling agent can burn through thousands of tokens before producing one user-facing answer — every tool description and response counts against the same context window as the conversation.
• Assuming greedy decoding is always safe. It's deterministic, which helps testing, but makes output more repetitive and prone to looping.
• Skipping the auth model on tool-calling setups. An LLM that can call tools is only as safe as what those tools are allowed to do.

Full article with the FAQ and quick summary: devtoolhub.com/what-is-an-llm