LLM Fundamentals Every Developer Should Actually Understand

If you're building anything with AI right now, you're probably treating the LLM like a magic box: you send text in, text comes out, and when it doesn't work you just... poke it differently and hope. That works up to a point. But there are a handful of underlying mechanics that, once you understand them, completely change how you debug prompts, design agents, and reason about cost and reliability.

I recently went through Matt Pocock's free "LLM Fundamentals" video series on AI Hero, and it's a genuinely great five-lesson crash course on exactly this. Below is my own write-up of the core ideas, partly as notes for myself and partly because I think every dev working with AI APIs should know this stuff cold.

1. Conversations are just a list of messages

Under the hood, a "chat" with an LLM isn't a chat at all — it's an array of messages that gets replayed to the model every single time. Two roles matter most:

• User messages — what you send
• Assistant messages — what the model sends back

Sitting above all of that is the system prompt: an instruction block at the very start of the history that's usually invisible to the end user but highly visible — and highly obeyed — by the model. If there's a conflict between what the system prompt says and what the user asks for, the model will generally side with the system prompt. This is the whole mechanism behind "custom GPTs," AI coding assistants with personalities, and most jailbreak-resistance efforts (imperfect as they are).

Two more things live inside assistant messages that are easy to overlook:

• Reasoning tokens — the "thinking" text some models produce before their final answer. It's not a separate channel; it's just another part of the same message.
• File parts — messages can carry more than text. You can attach a PDF for the model to summarize, or get an image back as part of a response.

The upshot: once you think of "the conversation" as a plain array of typed message parts instead of a chat bubble UI, a lot of AI SDK code stops feeling mysterious.

2. Text becomes numbers — and that's all a model actually sees

LLMs don't read words. They read tokens — numbers derived from your text via a process called encoding, and turned back into text via decoding.

The vocabulary of tokens is built by analyzing a huge corpus of text: start with individual characters, find common groupings of characters, then find common groupings of those groupings, until you've built up a vocabulary of a few tens of thousands of "chunks." A bigger vocabulary means common words get compressed into fewer tokens, which is more efficient — a word like "understanding" might take 5 tokens in a small vocabulary but only 2 in a large one.

This is also why unusual or made-up words get chopped up strangely — a nonsense word like "Frabjous" doesn't appear often (or at all) in the training corpus, so the tokenizer has to break it into a surprising number of small fragments instead of recognizing it as a clean chunk.

Why does this matter practically? Because:

• You're billed per token, input and output, usually at different rates.
• Input tokens include your entire conversation history, your system prompt, and your tool definitions — not just the message you just typed.
• Trimming unnecessary output (by asking for concise responses, structured formats, etc.) is a legitimate way to cut cost.

If you want to actually see this happen, the Tiktokenizer playground is a great way to watch text get chopped into tokens live.

3. The context window has a ceiling — and a soft spot

The context window is input tokens + output tokens, combined, and every model has a hard limit on it. Push a conversation past that limit and you'll get an error — sometimes mid-generation, where the model starts a response and simply runs out of room to finish it. The model has no built-in awareness of this limit; it can't gracefully plan around it.

But the harder problem isn't the hard limit — it's what happens well before you hit it: the "lost in the middle" effect. Information near the start and end of a long context gets attended to reasonably well; information buried in the middle gets less attention from the model, and retrieval quality drops accordingly. This gets worse, not better, as context windows get larger.

The practical implication is important: a huge context window is not a green light to dump everything you have into the prompt. A model that supports 1M tokens of context will usually still perform better with a focused 20K-token prompt than a sprawling 500K-token one. Bigger context is a safety net, not an invitation.

4. Tools are just a structured way for the LLM to ask you for things

"Tool calling" (or "function calling") sounds fancier than it is. Here's the whole mechanism:

1. You describe available tools in the system prompt — a name, a description, and a set of parameters defined via JSON Schema.
2. The user sends a message, e.g. "write a new file called .gitignore." You don't have to explicitly say "use the writeFile tool" — the model decides for itself which tool fits, if any.
3. The model responds not with plain text, but with a tool call: a structured message containing an id, the tool name, and the arguments it wants to call it with.
4. Crucially, nothing has actually happened yet. The model has only produced a message describing what it wants to do. Your application code is responsible for actually executing that action — writing the file, hitting the API, querying the database, whatever the tool represents.
5. You send a tool result back to the model, referencing the same id, describing what happened (success or failure — error messages need to go back to the model too, so it can adapt).
6. The model sees the full updated history and responds with a natural-language summary — something like "Done. What should go in there?"

That's it. Tool calling is a request/response loop layered on top of the same plain message array from section 1. This simple loop — describe tools, let the model choose, execute on your machine, report back — is the entire mechanism behind how tools like Claude Code and Cursor read files, run commands, and modify your codebase.

5. Agents vs. workflows: who's driving?

Once you're chaining multiple LLM calls together, you're building either a workflow or an agent — and the difference comes down to one question: who decides when to stop?

• Workflow: the developer writes the steps in code. Call the LLM, do something with the result, maybe call it again, then stop — all according to predetermined logic. The control flow lives in your code, not in the model's head.
• Agent: you hand the LLM a set of tools and let it decide, turn by turn, what to call next and when it's actually finished. There's no predetermined path — the model improvises its way through the problem.

Neither is strictly better. Agents shine when the steps required to solve a problem aren't knowable in advance and the system needs room to improvise. Workflows shine when the task is well-understood and needs to be done the same reliable way every time — and they're honestly underrated, since "agent" gets all the hype while "workflow" sounds boring by comparison.

A good example of a workflow pattern: splitting a long document into chunks, summarizing each chunk independently and in parallel, then summarizing the summaries. No improvisation needed — just orchestration. That's a workflow doing a job an agent would do less predictably and at higher cost.

Why this actually matters

None of these five ideas are complicated on their own. But together they explain almost every weird failure mode you'll hit building with LLMs:

• Model "ignoring" your instructions later in a long chat? Lost in the middle.
• API call failing on an otherwise-fine prompt? You blew the context window.
• Costs creeping up unexpectedly? Check your tool definitions and conversation history — they're input tokens too.
• Model doing something bizarre with an unusual identifier or made-up term? Tokenization artifacts.
• Agent looping forever or doing something unpredictable? You probably wanted a workflow.

Full credit to Matt Pocock for putting this together — the original five-part series has diagrams for every concept above and is worth watching directly if you want the visual walkthrough rather than my recap.

What's the LLM quirk that's bitten you the hardest in production? Curious to hear war stories in the comments.