DEV Community: Internals Decoded

Why Your Prompts Fail (and the Anatomy That Works)

Internals Decoded — Sun, 06 Sep 2026 16:21:26 +0000

Most prompts fail not because you forgot a magic phrase but because you are driving a probabilistic sequence model with a brittle text interface. The prompt becomes tokens, gets embedded, passes through attention layers with position dependent biases, and decodes token by token under sensitivity to phrasing, placement, and context length. The fix is a structured skeleton: role, task, constraints, and format.

Even when you phrase two prompts to mean exactly the same thing, the model can give you different error rates simply because the tokens split differently. A tiny change in whitespace can shift token boundaries, alter the positional vectors that govern attention decay, and send your instruction deep into the context’s dead zone where the model barely sees it.

How does an LLM actually process a prompt?

The model sees your text not as characters but as a sequence of token IDs that map to vectors in a high dimensional space, with positional encodings added to represent order. The entire prompt, including system messages and conversation history, is flattened into a single autoregressive sequence that feeds the transformer stack.

Think of a camera lens focusing light onto film. The lens is the embedding and positional encodings; the film is the attention layers. Changing the order of objects in the scene changes what the camera captures. Your prompt’s position and formatting determine which parts the model focuses on.

Tokenization is the first step. A learned tokenizer like BPE splits the text into subword units, mapping frequent character sequences to discrete IDs. This step is lossy and opaque. A small edit can change the token sequence length and boundaries, which shifts every downstream computation tokenization overview. Next, each token ID looks up an embedding vector from a learned matrix. That vector captures semantic and syntactic information from co-occurrence statistics. On top of the token embedding, the model adds a positional encoding. In modern models that use RoPE, the positional component applies complex rotations to the embedding, creating a smooth relationship between token distance and attention patterns positional vectors paper. Early tokens become strong positional anchors that influence the positional vectors of everything that follows.

flowchart LR
    A[Prompt text] --> B[Tokenizer]
    B --> C[Token IDs]
    C --> D[Embedding lookup]
    D --> E[Token vectors + positional encodings]
    E --> F[Transformer layers]
    F --> G[Logits for next token]
    G --> H[Sample token, repeat]

The transformer layers then mix these vectors with multi-head self-attention. Each head computes attention weights as scaled dot products between queries and keys, and different heads can specialize on delimiters, structure, or long range dependencies Causal Head Gating. Because future tokens are masked, the representation at any position depends only on earlier positions. Once your prompt is inside this machinery, every design choice about order, length, and delimiters becomes a pattern that attention heads either exploit or mishandle.

The entire pipeline explains why placement is not a cosmetic detail. Next we will examine the positional bias directly.

Why does placement of instructions matter so much?

Models pay disproportionate attention to tokens at the beginning and end of the sequence, and information in the middle suffers from a U-shaped performance drop. The earliest tokens form anchors that decay in influence as distance grows unless you refresh them.

Research on long context models found a consistent U-curve. Moving key information from the edges of the context into the middle can reduce question answering accuracy by more than thirty percentage points, even when the total input stays within the model’s nominal window U-shaped attention paper. The mechanism is rooted in RoPE based attention. The dot product between queries and keys for distant positions becomes less sensitive, especially once the distance exceeds what was typical during training Rotary Position Embeddings. Initial tokens create strong positional anchors, and their influence decays. The tokens near the very end benefit from recency bias because they are closest to the current generation point.

{
  "type": "bar",
  "title": "Instruction Accuracy by Position",
  "caption": "Illustrative data: performance drops when key instructions fall in the middle.",
  "data": [
    {
      "label": "Start",
      "value": 95
    },
    {
      "label": "Early",
      "value": 92
    },
    {
      "label": "Middle",
      "value": 70
    },
    {
      "label": "Late",
      "value": 88
    },
    {
      "label": "End",
      "value": 90
    }
  ]
}

Imagine our customer support reply assistant with a long system prompt. It defines brand voice, reply guidelines, escalation rules, and a rule that every reply must include the refund policy link. If that rule sits in the middle of a 2,000 token system prompt, and the customer’s query appears at the end, the model frequently ignores it. The assistant cheerfully answers without the link, because the token “policy link” received almost no attention from the final generation step. Moving that rule to the start of the system prompt and repeating it one sentence before the model must produce output repairs the failure. The anatomy that works puts critical instructions at both ends. This is the “start and end” pattern now recommended by official prompt guides OpenAI Prompt Engineering Guide.

Instruction drift is the same phenomenon in a longer conversation. When earlier system constraints are pushed deep into the history, they enter the low attention middle and silently stop influencing the output. The model did not forget. The tokens just became invisible to the attention mechanism.

Understanding where instructions lose their grip sets the stage for cataloguing the concrete failure modes that engineers encounter day to day.

What are the most common failure modes in prompts?

At production scale, prompt failures cluster into six categories from a recently published taxonomy: specification and intent, input and content, structure and formatting, context and memory, performance and efficiency, and maintainability and engineering defects Prompt Defects Taxonomy. Each category reflects a specific mismatch between the prompt's structure and the model's mechanics.

A specification defect appears when the prompt says “reply helpfully” but never defines what helpful means in that channel. Our assistant once generated a 500 word empathetic reply for a Twitter customer complaint that only accepted 280 characters. The intent was right but the constraint was missing. Input defects happen when retrieved documents conflict. A RAG (retrieval-augmented generation) pipeline fed an outdated refund policy alongside the question, and the assistant cited the wrong policy with high confidence because the most recent training data it had was the prompt’s own polluted context.

Structure defects emerge when delimiters are missing. Without clear separators, attention heads cannot segment the input into instructions, examples, and user content. The assistant responded to an old message from the chat history because the developer placed everything inside a single block with no markers. A context defect occurs when the total sequence exceeds the token window and silent truncation drops early system instructions. The assistant lost the rule “always verify account identity,” proceeded without it, and nobody noticed until a security audit.

A performance defect is a prompt that loads 10,000 tokens of examples on every call, driving up latency and cost. The model works, but the system cannot scale. A maintainability defect is a hardcoded prompt string in the backend code with no version and no tests. An edit to fix one edge case broke ten others, and the regression was discovered only through customer complaints.

These categories provide a diagnostic lens. Instead of treating “the model acted weird” as a black box event, you map the symptom to a defect type and apply a structured mitigation. This is the discipline that turns prompting from folk craft into engineering.

Before we can apply that discipline, we need to confront two subtle mechanics that cause failures even when the prompt’s logic is sound: tokenization and truncation.

How do tokenization and truncation cause hidden failures?

Tokenization is opaque and brutally sensitive. The same text can split differently depending on a leading space or a stray punctuation mark, altering the token count and the attention landscape. Truncation then silently deletes tokens from the start or middle of the context when the combined input overflows the window, leaving no error signal for the caller.

The phrase “customer support” may split into tokens like customer and support (with a leading space) in one case, or customer, -, support in another if a dash is present. These differences shift the positions of all subsequent tokens by one or two slots. If the token budget is tight, that shift can push a critical instruction out of the window entirely. The runtime drops the oldest tokens, so the assistant’s core identity disappears, and the model falls back to its training prior. Yet no error is surfaced. The response arrives with the usual status code and a normal looking answer that violates policy OpenAI Token Usage.

Detection requires active monitoring. You can check the token usage field in the API (application programming interface) response and correlate it with the expected prompt length. A finish reason of “length” means the output was truncated, but input truncation is silent. So you must log token counts per request and alert whenever usage bumps against the model’s limit. Without these checks, you will debug truncation failures as if they were reasoning failures, wasting days.

Tokenization and truncation turn a logically perfect prompt into a broken sequence. The same structural fragility extends to the invisible layers of instructions that the platform injects. That is the next piece.

How does the instruction hierarchy affect what I can control?

The APIs enforce an instruction hierarchy where system messages are designed to override user messages, and providers often add their own hidden system prompts. Your prompt is not the only set of instructions the model sees. Those hidden layers can bend behavior in ways that feel arbitrary from the outside.

Set a system prompt for the assistant: “you are a polite Acme agent, never mention competitors, always include the help link.” The provider may prepend its own system message about safety that forbids generating any commercial content. The combination can make the model refuse to answer a customer’s product question. You never see that hidden layer, so the refusal appears to come from nowhere. Research confirms that system prompts are not just another entry. They shift representational and allocative biases, and they interact in unpredictable ways when stacked System Prompt Biases Study. Because they sit at the very start of the sequence, they enjoy the positional advantage of the beginning, which makes them extremely influential.

The hierarchy also opens the door to prompt injection. A malicious user might inject text that mimics a system role marker and attempts to override your instructions. Defenses exist. You can tell the model to treat user content as untrusted and to ignore any text that claims to be a system directive. But because every word ends up as tokens in the same flat sequence, these defenses are not airtight. The safest strategy is to treat the LLM (large language model) output as untrusted input to later validation steps, a pattern that OWASP’s LLM Top Ten explicitly recommends OWASP Top 10 for LLMs.

Awareness of the hierarchy means you stop assuming full control and start designing prompts that are robust to partial overrides. The skeleton that follows does exactly that.

What does a reliable prompt structure look like?

A reliable prompt skeleton has four explicit parts: a role that aligns the model with the intended persona, a concrete task describing exactly what to produce, constraints that define allowed tone, length, and actions, and a format that specifies the output schema. You place the role and core rules at the start. You place the task, constraints, and format at the end, with a short reminder of any critical rule right before the expected output.

Here is how the skeleton transforms our customer support assistant. The original naive prompt:

You are a support assistant. Answer customer questions.

This prompt fails because the model has free reign to hallucinate persona, length, and policy. The structured version:

Role: “You are a customer support agent for Acme Corp. You follow these policies: never promise refunds over $50 without manager approval, always include the help center link, and remain empathetic.”
Task: “Given the customer email below, draft a reply that addresses their issue by referencing relevant policy and escalating if needed.”
Constraints: “Keep replies under 150 words. Do not use the competitor name ‘Globex’. If the issue is a billing dispute, start with an apology and state the escalation timeline.”
Format: “Reply in plain text, with a subject line on the first line and the body on following lines. Start the body with ‘Hello [Customer Name]’.”

In an API call, you put the role and core policies in the system message at the top. You put the task, constraints, and format in the user message, with the billing dispute rule repeated in one short sentence just before the model is expected to respond. This placement exploits the positional

{
  "type": "comparison",
  "title": "From Naive to Structured Prompt",
  "caption": "A structured prompt defines role, task, constraints and format to reduce ambiguity.",
  "before": {
    "label": "Naive prompt",
    "points": [
      "Vague persona",
      "No constraints",
      "No output format"
    ]
  },
  "after": {
    "label": "Structured prompt",
    "points": [
      "Explicit role",
      "Concrete task and constraints",
      "Defined output format"
    ]
  }
}

Originally published at Internals Decoded. AI internals, explained conversationally.

How to Actually Choose a Model

Internals Decoded — Fri, 04 Sep 2026 16:54:51 +0000

Choosing a model is a multi-objective decision. You weigh accuracy, latency, cost, context length, and safety against the hard constraints of your specific task. No single benchmark or leaderboard ranking can make this call for you. The process that works in production is: define your task precisely, build a representative evaluation dataset, run structured offline experiments to prune candidates, then validate the survivors with real users.

Here is the surprising part: the largest, most expensive model is rarely the right choice. Small fine-tuned models often beat generalist giants on specific tasks while being two orders of magnitude cheaper. Most teams waste money chasing marginal accuracy gains that users never notice.

What Does "Choosing a Model" Actually Mean?

Choosing a model is not one decision. It is a chain of decisions that starts with your task and ends with a running system.

graph TD
    A["Define task"] --> B["Build evaluation data"]
    B --> C["Filter by constraints"]
    C --> D["Compare on Pareto frontier"]
    D --> E["Validate with shadow deployment"]
    E --> F["A/B test"]
    F --> G["Monitor"]

The first decision is architectural. Do you need retrieval augmented generation (RAG) to ground answers in documents? Do you need fine-tuning to enforce a specific format or style? Your task's requirements might force a particular architecture before you even look at specific models.

Remember our running example of asking a chatbot for a recipe. If that chatbot needs to pull ingredients from a frequently updated company database, RAG becomes nearly mandatory. You need fresh data and source citations. This architectural choice shapes every model decision that follows. You now need an embedding model, a retriever, and a generation model that plays nicely with retrieved context.

In classical machine learning, model selection meant picking between a linear model and a tree ensemble. In the LLM world, you might choose between a tiny fine-tuned encoder and a massive generalist model. You might decide to route simple queries to a fast cheap model and complex queries to a slow expensive one. The decision space is larger now, but the core principle is unchanged: narrow the space using constraints before you compute a single metric.

How Do You Define Your Task Precisely Enough to Choose?

A vague goal like "make a good chatbot" is useless for model selection. You need a specification concrete enough to measure against.

Write down your task type. Is it classification, retrieval, generation, or reasoning? List your evaluation objectives. Are you optimizing for accuracy, factuality, helpfulness, or some business metric like conversion rate? Then list your hard constraints. Maximum tail latency in milliseconds. Minimum throughput in requests per second. Budget per request. Data residency requirements.

Suppose your recipe chatbot has a strict service level objective: p99 latency under two seconds. Any model that cannot meet that constraint gets eliminated immediately. You do not need to run a single benchmark to cross it off the list. Constraints define the feasible region. Everything outside that region is irrelevant.

This matters because engineers often start by browsing leaderboards. That is backwards. Start with your own requirements. The leaderboard comes later, as a filter on the candidates that already fit your constraints.

How Do You Build Evaluation Data You Can Trust?

You need a dataset that looks like the actual traffic your model will handle. Not a random sample of tweets. Not a general benchmark. Your data.

Teams typically assemble a "golden dataset" of a few hundred to a few thousand real queries paired with ideal responses. These queries come from production logs, from synthetic generation reviewed by experts, or from both. The dataset must span real user intents, input lengths, difficulty levels, and edge cases. Happy paths alone will mislead you.

For tasks with clear right answers, evaluation is straightforward. You run the model on each query and check if the output matches the expected answer. Accuracy or exact match works here.

For open-ended generation, things get harder. There is no single correct recipe response to "what can I cook with chicken and rice." You need qualitative judgments: helpfulness, clarity, safety. You can build a rubric and have humans score outputs. Or you can use a strong model as a judge, giving it the query, the output, and a scoring rubric. This scales better but requires careful prompt design.

The crucial mechanical detail: keep your evaluation data separate from anything you use to tune the model. If you tweak prompts based on test set performance, you have contaminated your test set. You need a held-out set that sees zero exposure until final evaluation. Standard three-way splits (train, validation, test) still apply here. For LLM work where you are not training, reserve some evaluation data for hold-out testing of prompts, routing rules, or RAG parameters.

What Do Benchmarks Like MMLU and TruthfulQA Actually Tell You?

Static benchmarks are diagnostic tools. They are not final arbiters of model quality for your task.

MMLU (Massive Multitask Language Understanding) tests knowledge across dozens of subjects through multiple-choice questions source. A high MMLU score means broad factual knowledge. It does not mean the model writes good recipes or follows formatting instructions. TruthfulQA tests whether models avoid repeating common misconceptions source. A model that scores well here is less likely to tell you to put glue on pizza because it read that somewhere online. ARC tests science reasoning. HellaSwag tests commonsense continuation selection.

These benchmarks are attractive because they are automated. You download the dataset, run your model, and get a number. But they have limits. Models can be over-optimized for public benchmarks through training or prompt engineering. A two-point difference on MMLU between two models often means nothing for your specific task. And none of these benchmarks measure latency, cost, or how well a model handles your company's internal jargon.

Use benchmarks as sanity checks. If a model cannot handle basic commonsense reasoning on HellaSwag, it probably should not power your customer-facing chatbot. But do not use benchmark rankings as your primary selection criterion. Your golden dataset matters more.

How Do You Compare Models When There Are Multiple Objectives?

You cannot reduce model selection to a single number. Accuracy, latency, cost, and context length are all real constraints. The model that wins on accuracy might lose catastrophically on latency.

Pareto frontier analysis gives you a clear framework. Plot your candidates. A model is dominated if another model beats it on every metric. Eliminate all dominated models. What remains is the Pareto frontier: the set of models where improving one metric means sacrificing another. Decision makers then pick from this frontier based on their trade-off preferences.

You might formalize this with a composite metric like the Performance Efficiency Ratio (PER), which combines accuracy, throughput, memory usage, and latency into one normalized score source. Small models in the 0.5 to 3 billion parameter range often dominate on PER, especially in resource-constrained deployments. They give up a few points of accuracy and gain orders of magnitude in speed and cost.

Total cost of ownership per token is another composite worth computing. It combines hardware capital cost, operational cost (power, cloud), and engineering maintenance cost, divided by total lifetime tokens served. The cheapest GPU per hour is not always the cheapest deployment per token. Throughput and utilization matter more.

For our recipe chatbot, you might measure accuracy on your golden dataset of recipe requests, p99 latency at peak load, and cost per thousand requests. A small fine-tuned model might score 87% accuracy at 200ms p99 latency and cost $0.001 per request. A giant generalist model might score 91% at 800ms p99 latency and cost $0.05 per request. Users rarely notice the 4% accuracy gap. They absolutely notice a half-second delay. The frontier makes this trade-off explicit.

{
  "type": "stat",
  "title": "Recipe Chatbot Metrics",
  "caption": "Illustrative numbers for a small fine tuned model serving recipe requests.",
  "stats": [
    {
      "value": "92%",
      "label": "Accuracy on golden dataset"
    },
    {
      "value": "1.8s",
      "label": "p99 latency"
    },
    {
      "value": "$0.03",
      "label": "Cost per 1k requests"
    },
    {
      "value": "8k",
      "label": "Context window tokens"
    }
  ]
}

How Do You Validate a Model Before Full Deployment?

Offline evaluation on your golden dataset is necessary. It is not sufficient. Users behave differently than evaluation datasets predict.

Shadow deployment is the safest next step. Run the candidate model in parallel with your current production model. Send it the same real inputs. Log its outputs. Compare them against the incumbent. Users see no change. You collect real-world data on how the candidate behaves under live traffic.

When you are confident enough to expose users, run an A/B test. Route a small percentage of traffic to the candidate. Measure real outcomes: user satisfaction ratings, task completion rates, whether people immediately re-ask the question in a different way (a strong signal the first answer was bad). Statistical analysis tells you whether any observed differences are real or noise.

Monitor continuously after full deployment. User behavior drifts. Input distributions shift. The model that worked in January might degrade by June. Log inputs and outputs. Periodically score samples using your evaluation pipeline. Set alerts for metric degradation. This is not a "choose once" problem. It is an ongoing process.

Quick Reference: Key Decision Factors

Factor	What to Measure	Why It Matters
Task fit	Accuracy on your golden dataset	General benchmarks do not predict domain performance
Latency	p50 and p99 response time	Users feel delays above 200ms. Strict SLOs may force smaller models
Cost	Total cost per request or per token	Marginal accuracy gains often cost 10-50x more
Context window	Tokens needed for your typical input plus retrieved documents	A model with a 4K window fails on tasks needing 8K tokens
Safety and governance	Hallucination rate, refusal rate, data residency	A model that makes things up confidently destroys user trust
Maintainability	Engineering effort to deploy, monitor, and update	Open models give control but add operational burden

Frequently Asked Questions

Q: Should I just use the model at the top of the LMSYS leaderboard?

No. The LMSYS Chatbot Arena measures human preference across arbitrary prompts. It correlates weakly with performance on your specific task, tells you nothing about latency or cost, and reflects a generalist use case that may not match yours. Use it as a discovery tool to find candidates, not as a final decision criterion.

Q: When does a small fine-tuned model beat a large general model?

When your task is narrow and well-defined. Classification, entity extraction, structured output generation, and domain-specific Q&A on a stable corpus all favor small fine-tuned models. A 7B parameter model fine-tuned on your data can match or beat a generalist 70B model on your specific task while being faster and cheaper by orders of magnitude source.

Q: How many evaluation examples do I need?

Enough to cover your real input distribution and edge cases. For most production applications, 300 to 1,000 representative examples provide a reliable signal. Too few examples and your variance is high. Too many and you burn time and money on evaluation without gaining confidence. Start small, measure variance, and add examples where you see high uncertainty.

Q: Can I trust LLM-as-a-judge evaluations?

Partially. A strong judge model with a clear rubric correlates reasonably well with human judgments. But judge models carry biases. They tend to favor longer responses and their own model family's outputs. Validate your judge against human annotations on a subset of your data before relying on it. For high-stakes decisions, keep humans in the loop.

Q: How often should I re-evaluate my model choice?

Continuously monitor. Re-evaluate formally when you see metric degradation, when significantly better models become available, or when your task requirements change. For most teams, a quarterly structured re-evaluation alongside ongoing monitoring strikes the right balance between stability and improvement.

Test Yourself

Your team is building a customer support bot that answers questions from a 10,000-page product manual. The manual changes weekly. Your budget caps inference cost at $200 per month. Latency must stay under one second. A teammate proposes using the latest 405B parameter model because it tops the MMLU leaderboard. Another suggests fine-tuning a 7B model on a snapshot of the manual. Which approach is likely to fail and why?

Answer: The 405B model approach fails on cost and freshness. At typical API pricing, serving even a few thousand queries per day with a 405B model would blow past the $200 monthly budget within days. The fine-tuning approach fails on freshness. A model fine-tuned on a snapshot of the manual becomes stale the moment the manual updates. Weekly fine-tuning runs are engineering-intensive and test your ability to ship model updates reliably. The correct approach here is RAG with a small, fast model. You embed the manual chunks in a vector database, retrieve relevant sections for each query, and feed them as context to a 7B or even 3B model. This keeps answers grounded in the current manual version, stays within the latency budget (small models are fast), and costs far less than serving a giant model. You update the vector database weekly without retraining anything.

{
  "type": "comparison",
  "title": "Model Choice: 405B vs Small Fine Tuned",
  "caption": "For a customer support bot with a frequently updated manual, the small fine tuned model wins on cost, latency, and freshness.",
  "before": {
    "label": "405B general model",
    "points": [
      "High API cost",
      "High latency",
      "Cannot update weekly"
    ]
  },
  "after": {
    "label": "Small fine tuned model",
    "points": [
      "Low cost",
      "Low latency",
      "Weekly fine tuning possible"
    ]
  }
}

If you want this kind of breakdown every week — not hype, not leaderboard worship, but how real systems actually work under the hood — subscribe to Internals Decoded at internalsdecoded.com.

Sources

Originally published at Internals Decoded. AI internals, explained conversationally.

Fine-Tuning, Explained: Teaching an Old Model New Tricks

Internals Decoded — Wed, 02 Sep 2026 17:13:41 +0000

Fine-tuning takes a pre-trained model and subjects it to a second round of training on a small, task-specific dataset. The result is a permanent behavior change: a generic chatbot learns to write in your brand voice, a language assistant starts composing emails exactly the way your team does, a medical support bot adopts the shorthand of clinicians. Unlike prompting, which wraps a suggestion around the model, fine-tuning rewires the model’s internal weights so the new behavior becomes the default.

But here is the paradox that catches most teams off guard: For the vast majority of customization problems, you should not fine-tune at all. You should use retrieval-augmented generation (RAG) instead. RAG is cheaper, safer to roll back, and immune to the most damaging side effect of fine-tuning: catastrophic forgetting. The moment you really need fine-tuning, when you must change how the model speaks, not what it knows, no other technique will do.

[In the last episode we compared open and closed models, examining weights, licenses, and the control each grants you. Today we move from “what model” to “how do I make that model do exactly what I want?”]

What actually happens inside the model when you fine-tune?

Imagine a chef who trained in every culinary tradition on earth. She knows how to sauté, braise, ferment, and flambé, but she has never worked in your kitchen. Your restaurant serves a handful of signature dishes with very specific techniques. You could give her a checklist every morning (a prompt), but she might still occasionally slip back into her classical training. Fine-tuning is the equivalent of putting her through a week-long boot camp in your kitchen: you hand her the 40 dishes that define your menu, have her cook them over and over, and by Friday she no longer needs the checklist. The recipes are now muscle memory.

Technically, the model starts as a set of parameters θ₀ learned during pretraining on trillions of tokens. Those parameters already encode the basics of language, common sense, and some world knowledge. You then collect a dataset of input-output pairs that represent the behavior you want. For a chatbot that drafts emails in your company’s tone, that dataset might be 200 pairs of rough notes (input) and the corresponding polished emails (output). During fine-tuning, the model sees each input and tries to predict the output token by token. A loss function measures how far off the prediction was. Gradients flow backward from that loss, nudging the weights just enough to make the model’s output distribution line up with your examples.

Because the starting point is already a fluent language model, these updates are tiny and need not run for many steps. You typically use a learning rate one or two orders of magnitude smaller than during pretraining. The dramatic result is that after seeing only a few hundred examples, the model internalizes a pattern, the exact wording style, the preferred format, the domain-specific jargon, and reproduces it reliably, without explicit instruction.

Which fine-tuning methods exist, and when do you use them?

The phrase “fine-tuning” actually covers several distinct techniques. The simplest is supervised fine-tuning (SFT). You give the model input-output pairs and train it to produce the output verbatim, using a cross-entropy loss on the completion tokens. This works beautifully when the desired behavior can be captured in examples: “When you see a bullet list of facts, turn it into a three-sentence summary in the voice of a friendly physician.”

When the goal is to follow natural-language instructions without needing a strong template, instruction tuning enters the picture. Instruction tuning is just SFT on a dataset built entirely of diverse instructions and human-written responses. The original pre-trained model learned to predict next words; instruction tuning teaches it that sequences like “Summarize the following article: ...” are commands, not just co-occurring text. Research shows that a few thousand well-chosen instruction-response pairs can turn a chaotic base model into a helpful assistant source source.

Full-model fine-tuning updates every weight. That gives the strongest adaptation but demands a lot of memory and storage: a 7-billion-parameter model in 16-bit precision already eats 14 GB just for the weights, and you need extra room for gradients and optimizer states. Parameter-efficient fine-tuning (PEFT) avoids that cost. LoRA (low-rank adaptation), the most popular PEFT method, freezes the entire base model and injects small trainable matrices that capture the “delta” needed for the new task. During inference, you can merge those tiny matrices back into the base weights, so the fine-tuned model runs at exactly the same speed as the original. LoRA typically updates less than 0.1 % of the parameters, dropping memory consumption by up to 3× versus full fine-tuning while holding performance close to par source source.

{
  "type": "comparison",
  "title": "Full model fine tuning vs LoRA",
  "caption": "Memory needed for a 7B parameter model in 16 bit precision. LoRA updates less than 1% of weights.",
  "before": {
    "label": "Full model fine tuning",
    "points": [
      "Updates 7 billion weights",
      "Requires ~14 GB GPU memory",
      "Stores a full model copy"
    ]
  },
  "after": {
    "label": "LoRA fine tuning",
    "points": [
      "Updates ~50 million weights",
      "Requires ~1 GB GPU memory",
      "Stores a tiny adapter file"
    ]
  }
}

A final category, alignment fine-tuning, optimizes for human preferences rather than exact output strings. The most famous pipeline is RLHF (reinforcement learning from human feedback), a multi-stage process that collects preference judgments, trains a reward model, and then uses reinforcement learning to push the language model toward helpful, harmless responses. A simpler alternative, DPO (Direct Preference Optimization), folds the preference learning directly into a supervised-style loss, eliminating the need for a separate reward model or reinforcement learning loop. DPO often matches RLHF’s alignment quality with far less engineering overhead source. You use alignment tuning when the problem is not “what answer” but “what constitutes a good answer”, safety, politeness, refusal of harmful requests.

When does fine-tuning actually help, and when does RAG beat it?

The decision between fine-tuning and retrieval-augmented generation comes down to one question: Are you trying to change what the model knows, or how the model behaves?

If you need the model to incorporate up-to-date facts, the latest product prices, today’s weather, the contents of your private knowledge base, RAG is almost always the right answer. With RAG, you leave the model untouched and inject relevant documents into the prompt at query time. The model reads them and bases its answer on them. You can update the documents without retraining, you get built-in provenance (the model can cite the source), and you avoid any risk of degrading general performance. The running-example chatbot that helps with recipes is a perfect candidate for RAG: the model’s language skills stay general, but each request pulls a fresh list of ingredients and steps from your database.

Fine-tuning shines when you need to alter the model’s intrinsic style or format. Suppose your recipe bot must always suggest vegan substitutions and present ingredients in a table, no matter how the user phrases the request. You could try to force this with a long system prompt, but the model might occasionally forget the formatting, especially in longer conversations. Fine-tuning on 300 curated examples of ideal responses will bake that behavior into the model so deeply that prompting becomes secondary. Similarly, if your team’s internal chatbot must adopt a specific corporate voice, “concise, never start a sentence with ‘I think’, always use bullet points for steps”, a small supervised fine-tuning run will produce consistent, effortless compliance.

{
  "type": "stat",
  "title": "When fine tuning helps",
  "caption": "Style and format changes need far fewer examples than teaching new knowledge.",
  "stats": [
    {
      "value": "200 to 500",
      "label": "Examples for tone and format"
    },
    {
      "value": "1000 to 10000",
      "label": "Examples for instruction following"
    },
    {
      "value": "Not recommended",
      "label": "For factual knowledge (use RAG)"
    }
  ]
}

A real-world hybrid also works: keep the knowledge side in RAG, fine-tune only the style layer. A LoRA adapter trained on tone and format sits on top of a frozen base model, while the retrieval system supplies factual content. This separates the concerns cleanly: RAG handles what is said, fine-tuning handles how it is said.

Why most teams never need fine-tuning

The hype around fine-tuning creates the impression that customizing a model means retraining it. In practice, good prompt engineering and a solid RAG pipeline cover 80 % of enterprise AI use cases. Fine-tuning introduces burdens that many teams underestimate.

First, data. You need a high-quality, well-curated dataset that represents exactly the behavior you want. A few messy examples from a Slack channel will not do. Cleaning, labeling, and validating that dataset is manual work that often takes longer than building the retrieval pipeline.

Second, evaluation. Prompt tweaks can be A/B tested and rolled back instantly. A fine-tuned model is a new artifact that must be benchmarked not just on the target task but on everything the model used to do well. Did it forget how to handle multi-turn conversations? Is it more likely to hallucinate numbers? Finding out requires a broad evaluation suite that most teams do not have off the shelf.

Third, cost. While LoRA reduces the compute bill, fine-tuning still consumes GPU (graphics processing unit) hours, requires careful hyperparameter sweeps, and forces you to maintain and serve multiple model artifacts. By contrast, a retrieval index is cheap to build, cheap to update, and fits into the same inference pipeline.

Fourth, fragility. Because fine-tuning physically changes the model, a small mistake in the training data can introduce subtle, hard-to-detect biases. A model fine-tuned to be “empathetic” might start apologizing when asked factual questions, a drift that only appears in production after thousands of interactions.

For most teams, the pattern is: start with prompt crafting, add RAG when you need facts, and reserve fine-tuning for the narrow cases where style or format refuses to budge any other way. The recipe chatbot that merely fetches ingredients via RAG while keeping a friendly tone via a prompt often satisfies users completely. You would only fine-tune it if you needed a very specific output structure or a branded voice that prompts could not reliably sustain.

What is catastrophic forgetting, and how do you manage it?

Every fine-tuning step risks eroding the model’s general knowledge. This is catastrophic forgetting: the model becomes so specialized that it loses skills it previously possessed, sometimes dramatically. You might fine-tune a model on medical notes and discover it can no longer do basic arithmetic or handle a date correctly.

Empirically, forgetting is worse for larger models in the 1-to-7-billion-parameter range when trained on narrow distributions. One reason is that a finely-tuned model can settle into a sharp minimum of the loss landscape, a spot where moving the parameters even a tiny bit causes a steep increase in loss on other tasks. Small gradient steps for a new task can therefore throw it out of those high-performing regions completely source.

Several strategies combat forgetting. Rehearsal mixes a small amount of general data into each fine-tuning batch, reminding the model of its original distribution. Regularization methods like Elastic Weight Consolidation add a penalty that discourages changes to parameters deemed important for prior tasks. Parameter-efficient methods provide a different kind of protection: because LoRA freezes the base model and only updates a handful of new parameters, the original weights stay intact, preserving general capabilities by construction. This is one of the quiet reasons LoRA is popular, it naturally isolates the adaptation, so the base model’s broad knowledge remains untouched source.

In practice, when you fine-tune with LoRA at a small rank (8-16) on a focused style dataset, catastrophic forgetting is rarely a problem. The real risk emerges when you attempt full-model fine-tuning on very narrow tasks without any general-data rehearsal. Knowing this, a safe workflow is to start with LoRA, evaluate broad benchmarks after training, and only escalate to full fine-tuning if the style shift demands it and your evaluation shows no regression.

graph TD
  A[Base pretrained model] --> B[Apply LoRA adapter]
  B --> C[Train on narrow style data]
  C --> D[Model keeps general skills]
  D --> E[Catastrophic forgetting avoided]
  A --> F[Full model fine tuning]
  F --> G[Train on narrow data]
  G --> H[Model loses general skills]

Quick Reference

Property	Detail
Data needed for style/form change	100-1,000 curated examples
Data needed for instruction following	1,000-10,000 diverse instruction-response pairs
Primary objective (SFT)	Cross-entropy on response tokens
Primary objective (alignment)	DPO preference loss or RLHF reward maximization
Memory-efficient option	LoRA (rank 4-16, α = rank)
Typical learning rate	1×10⁻⁵ to 5×10⁻⁵ (vs 1×10⁻⁴ for pretraining)
Main risk	Catastrophic forgetting of general capabilities
RAG alternative	Cheaper, safer for factual updates and dynamic data
LoRA inference impact	Zero latency increase after merging weights

Frequently Asked Questions

Q: Can I fine-tune a model to “know” new facts, like my company’s product catalog?
No. Fine-tuning is inherently bad at memorizing factual propositions reliably; it optimizes for distributional patterns, not discrete database inserts. Store your catalog in a retrieval index and use RAG to feed the facts to the model at runtime. The rare case where you must bake knowledge into weights, for example, the exact spelling of your proprietary acronyms, can be done with a small LoRA run on a few hundred repeats, but even then RAG with a glossary is more auditable.

Q: How many examples do I really need to fine-tune for a tone and format change?
Anywhere from 200 to 500 high-quality examples often suffices for a consistent stylistic shift. The examples should be diverse in phrasing and cover edge cases you care about (short requests, long multi-step inputs, empty inputs). More examples buy resilience but with diminishing returns. Tools like the OpenAI fine-tuning dashboard show a loss curve that helps you detect when the model has saturated.

Q: Is fine-tuning expensive compared to prompt engineering?
In terms of initial setup, yes: you pay for GPU hours and data curation. In terms of ongoing inference, fine-tuning can actually reduce cost and latency because you drop the need for long, detailed system prompts and multiple examples in the context. A fine-tuned model that already knows your format needs only a short prompt, so each request uses fewer input tokens.

Q: What is the cheapest way to get started with fine-tuning?
Use a LoRA-based library such as Hugging Face PEFT with a quantized base model (4-bit loading). On a single A10G or even a high-end consumer GPU, you can fine-tune a 7-billion-parameter model on a few hundred examples in under an hour for a few dollars of cloud compute. Many cloud providers now offer managed fine-tuning endpoints that handle the infrastructure for you with straightforward API (application programming interface) calls.

Q: Do I need special serving infrastructure for a fine-tuned model?
No. Once you merge a LoRA adapter into the base weights, the fine-tuned model is architecturally identical to the original. You serve it from the same inference engine (vLLM, TGI, etc.) with no additional latency. If you want to maintain multiple task-specific heads without duplicating the base model, you can keep the adapters separate and swap them dynamically, which does require a serving stack that supports adapter loading.

Test yourself

Your team runs a medical Q&A chatbot for doctors. The bot uses RAG over a large clinical database and answers questions factually. Doctor users complain that the answers sound “like Wikipedia,” not like a colleague sharing a quick insight. They want the bot to use medical shorthand, drop unnecessary hedging, and start answers with the most actionable information. Prompt engineering slightly improved the tone but still feels robotic. Someone suggests fine-tuning. Would you fine-tune, and if so, how would you avoid breaking the factual reliability?

Answer: Yes, this is a tone and style problem exactly where fine-tuning pays off. You should fine-tune, but not on the whole model. Use a LoRA adapter with a small rank (8-16) trained on 300-500 examples of “ideal” doctor-to-doctor answers that mirror the desired voice, while keeping the RAG pipeline untouched. The LoRA adapter will shift the model’s generation style without touching the base weights, preserving its ability to read and synthesize retrieval content. To prevent any drift in factual accuracy, include a small set of factual questions with correct answers in the training set (rehearsal) and evaluate after each epoch on a held-out factual benchmark. If the LLM (large language model) ever starts hallucinating under the new style, dial down the LoRA rank or add an early-stopping trigger based on factual accuracy. This approach gives the tone you need while building a safety barrier against regression.

Where to go next

The customization puzzle has one more chapter: production deployment. In the final episode, we will pull together training, inference, context, and alignment to show how you ship a reliable AI system that tourists actually trust.

Originally published at Internals Decoded. AI internals, explained conversationally.

Open vs Closed Models: What 'Open Source AI' Really Means

Internals Decoded — Mon, 31 Aug 2026 19:32:57 +0000

When you ask a chatbot for a recipe, you're talking to a model. But what does it actually mean for that model to be "open source"? The term now means three completely different things depending on who's talking: the weights you can download, the data that trained it, and the license that governs what you can do with either. Most models marketed as open are missing at least one of these pieces.

Here's the part that surprises engineers: downloading the weights of a model like Llama does not make it open source. Not even close. The Open Source Initiative now has a formal definition, and most "open" models fail it on multiple counts. Understanding which parts are actually open determines whether you can audit a model for bias, reproduce its training, or embed it in a product without a lawyer.

What does "open source AI" actually require?

The Open Source Initiative defines Open Source AI as a system where you can use, study, modify, and share every component without asking permission. This means three concrete artifacts must be available under an OSI-approved license: detailed information about the training data, the complete code used for training and inference, and the model parameters including weights and checkpoints. source

For a machine learning system, the "preferred form for making modifications" is not just source code. It includes data provenance, collection methods, labeling procedures, and processing steps. A skilled person should be able to build a substantially equivalent system from what's provided. source

This is a much higher bar than most people realize. When a company releases model weights on Hugging Face and calls it open source, they are almost certainly leaving out the training data documentation and the actual training scripts. What they've released is open weights, not Open Source AI.

Why do people confuse weights with open source?

The confusion starts with how software engineers think about openness. In traditional open source, the source code is the thing. Download the repo, read the code, build it, run it. The artifact and the blueprint are the same thing.

Model weights are not source code. They are the output of a training process, a compressed representation of patterns found in the training data. Having the weights lets you run inference and fine-tune. It does not let you understand how the model was built, what data shaped it, or whether that data contained copyrighted material or biased content. source

Think of it like getting a compiled binary versus getting the source tree plus the build system plus the compiler flags. The binary runs. You can even patch it. But you cannot reproduce the build, audit the dependencies, or verify that nothing sketchy happened during compilation. Open weights are the binary. Open Source AI is the whole build pipeline.

{
  "type": "comparison",
  "title": "Weights vs. Full Open Source",
  "caption": "Open weights let you run and fine-tune, but open source AI gives you the full picture.",
  "before": {
    "label": "Open Weights",
    "points": [
      "Use the model for inference",
      "Fine-tune on custom data",
      "Deploy behind an API"
    ]
  },
  "after": {
    "label": "Open Source AI",
    "points": [
      "Retrain the model from scratch",
      "Inspect and modify training code",
      "Reproduce the exact training process",
      "Audit data lineage and labeling"
    ]
  }
}

What's actually in a model release?

When a company releases a model, you typically get a set of files. The weights file contains millions or billions of floating-point numbers representing what the model learned during training. A config file describes the architecture: how many layers, the hidden size, the attention mechanism. A tokenizer file maps between text and the token IDs the model actually processes.

What you almost never get is the training data manifest, the data filtering and deduplication scripts, the exact training hyperparameters, or the evaluation harness used to measure the model. These are the parts that would let someone reproduce or audit the training process. Without them, you're accepting the model as a black box that you can run but cannot fully understand.

Some releases go further. EleutherAI, for example, releases training datasets and codebases alongside weights, intentionally aligning with OSI principles. This is the exception, not the norm.

How do licenses reveal what's really open?

The license is the fastest way to cut through marketing language. An OSI-approved license like Apache 2.0 or MIT allows any use, any modification, any redistribution, with no field-of-use restrictions and no revenue thresholds. If a model license says you need a commercial license above a certain revenue level, it is not open source. source

Mistral AI illustrates this spectrum clearly. Some of their models use Apache 2.0, which qualifies as open source. Others use a modified MIT license requiring companies with monthly revenue above 20 million USD to obtain a separate commercial license. That revenue threshold is a field-of-use restriction. It violates OSI's requirement that open source licenses cannot discriminate based on user size or commercial activity.

Meta's Llama license goes further. It imposes usage restrictions and does not provide full transparency into training data. The OSI has explicitly stated this falls short of Open Source AI requirements. source xAI's Grok weights were released with a custom license containing anti-competitive terms. source Hugging Face supports gated models where access requires approval, which also violates the "no permission needed" requirement. source

How do regulators think about this?

Regulators are building their own categories, and they don't always align with OSI definitions. The EU AI Act uses the term "general-purpose AI models" (GPAI) for models trained with large amounts of data using self-supervision at scale. Recital 98 suggests models with at least a billion parameters trained this way should be considered GPAI. source

The Act distinguishes between GPAI models and GPAI models with systemic risk, attaching different obligations to each. Openness of weights matters mainly for documentation exemptions and risk management requirements. A model with widely available weights gets some regulatory relief, but the Act does not use "open source" as a formal category. source

In the US, the NTIA and related executive orders use "dual-use foundation models with widely available model weights" or "open foundation models." Again, the focus is on weight availability, not on OSI-style full openness. source This creates a gap: a model can be "open" for regulatory purposes while failing OSI's definition of Open Source AI.

Why would anyone choose closed?

Closed models accessed via API (application programming interface) centralize control. The provider manages inference infrastructure, safety filters, telemetry, and updates. For the user, this means no GPU (graphics processing unit) provisioning, no model serving code to maintain, and no weight files to secure. It also means the provider sees every prompt and response, which may be a feature or a dealbreaker depending on your data sensitivity. source

Closed providers can also iterate faster on safety. When a new jailbreak technique emerges, they can patch the API layer centrally. Open-weight model deployers must monitor and patch their own deployments. This is real operational overhead that teams often underestimate. source

The tradeoff is dependency. A closed model's capabilities, pricing, and availability are controlled by the provider. If the provider deprecates a model version or changes its acceptable use policy, you adapt or migrate. With open weights, you control the artifact and can run it indefinitely on your own infrastructure. source

Quick Reference

Property	Open Source AI	Open Weights	Closed Model
Weights available	Yes	Yes	No
Training code available	Yes	Usually no	No
Data documentation	Detailed	Minimal or none	None
License type	OSI-approved	Custom, often restricted	Proprietary
Can reproduce training	Yes	No	No
Can audit for bias	Yes	Partial	Limited
Can fine-tune freely	Yes	Usually yes	No
Can deploy commercially	Yes, unrestricted	Often has revenue limits	Via API only
Regulatory category (EU)	GPAI	GPAI	GPAI
Example	EleutherAI models	Llama, Mistral (some)	GPT-4, Claude

graph TD
A["Closed Model"] --> B["Open Weights"]
B --> C["Open Source AI"]
A -->|provides| A1["No public model files"]
B -->|provides| B1["Weights file only"]
C -->|provides| C1["Weights Code Data"]

Frequently Asked Questions

Q: Can I use Llama in a commercial product without paying Meta?

You can use the weights under the Llama license, which allows commercial use but imposes restrictions. It is not an OSI-approved license. Read the specific terms carefully. If your use case triggers the restrictions, you need a separate agreement with Meta. source

Q: What's the practical difference between Apache 2.0 weights and a custom license?

Apache 2.0 gives you perpetual, irrevocable rights to use, modify, and distribute the weights for any purpose. A custom license can add conditions like revenue thresholds, use-case restrictions, or attribution requirements. Custom licenses can also be changed unilaterally by the licensor for future releases. Read the grant clause. source

Q: Does "open weights" mean I can see what data the model trained on?

No. Open weights means you have the final trained parameters. Training data documentation is a separate artifact. Most open-weight releases provide only high-level descriptions of training data sources, not detailed manifests. Without that, you cannot fully audit for copyrighted material or bias.

Q: Are there any truly open source large language models?

Yes, but they are rare. EleutherAI's models are released with weights, training code, and dataset documentation under open source licenses. They intentionally meet OSI's definition. Most other models marketed as open source are actually open weights with custom licenses.

Q: How does the EU AI Act affect my choice between open and closed models?

If you deploy a GPAI model in the EU, you have transparency and documentation obligations regardless of whether it's open or closed. Models with widely available weights get some exemptions from detailed technical documentation requirements. The Act does not exempt open models entirely. Check the specific obligations for your deployment context. source

Test yourself

You're evaluating two models for a customer support chatbot that will handle sensitive financial queries. Model A is closed, accessed via API, with SOC 2 compliance and contractual data processing terms. Model B is "open source" according to its marketing, with downloadable weights under a custom license that prohibits use in financial services without a separate agreement. The training data provenance for Model B is described as "a mix of public web data and licensed corpora." Which model gives you better compliance posture, and why?

Answer: Model A gives you better compliance posture for this specific use case, despite being closed. The API provider offers contractual commitments about data handling and security certifications that you can present to auditors and regulators. Model B's custom license explicitly restricts financial services use, so you'd need a separate agreement anyway, negating the "free" aspect of open weights. More critically, the vague training data description means you cannot verify whether the model was trained on confidential financial data or copyrighted material, which creates IP indemnification risk. For regulated industries, contractual clarity and audit trails matter more than weight availability. If you wanted an open-weight option, you'd need a model with an OSI-approved license that doesn't restrict financial services and with detailed enough data documentation to satisfy your compliance team's due diligence requirements.

What's next

In the next episode, we'll look at fine-tuning: what actually happens to those weights when you train a model on your own data, and why it's both more powerful and more fragile than most people assume.

If you want this kind of breakdown every week (how real AI systems actually work under the hood, not marketing summaries), subscribe to Internals Decoded at internalsdecoded.com.

Sources

Originally published at Internals Decoded. AI internals, explained conversationally.

Why Models Make Things Up

Internals Decoded — Sat, 29 Aug 2026 17:15:46 +0000

Last time, we saw how a model's working memory, its context window, can get overloaded, causing long conversations to drift. This time we tackle the most infamous side effect of that same predictive engine: the model confidently inventing facts.

When a language model tells you that the Eiffel Tower is in Rome, it isn't lying. It's doing exactly what it was built to do: predict the next plausible word. Hallucination is not a separate failure mode. It's the same mechanism that makes the model fluent, creative, and useful, just operating where plausibility and truth have drifted apart.

Even a perfectly trained, well-calibrated model must hallucinate on rare facts. That's not an engineering oversight. It's a statistical inevitability baked into the training objective. Once you see why, the whole phenomenon stops feeling mysterious and starts looking like a predictable property of the system.

Why does a model that's so good at predicting words sometimes get facts completely wrong?

The model is not a knowledge base. It's a next-token prediction machine. When you ask it a question, it doesn't look up an answer. It generates a continuation that would be probable in the text it was trained on. If the most probable continuation is factually wrong, the model will confidently produce it.

Think of a chef who has only ever read cookbooks. She can write a flawless recipe for a dish she's never tasted. If the cookbooks contain a typo that calls for salt instead of sugar, her recipe will be confidently wrong. She isn't lying. She's reproducing the pattern she learned.

This is exactly how a language model works. During training, it sees trillions of words from the internet, books, and articles. It learns to minimize the difference between its predictions and the actual next tokens in that data. The objective is purely statistical: maximize the probability of the observed text. There is no separate signal for truth. The model never learns that "vaccines cause autism" is false; it only learns that this phrase appears in certain contexts alongside other phrases. When a prompt echoes those contexts, the model's distribution can assign high probability to the false statement source.

So hallucination happens whenever the model's learned distribution over continuations diverges from the distribution of factually correct answers. The model is optimizing for "what sounds right," not "what is right." That's the core of it.

What does the training data have to do with hallucinations?

The training data is the only window the model has onto the world. If a fact appears rarely, or appears in conflicting versions, the model cannot reliably learn it. This is not a flaw in the data cleaning pipeline. It's a fundamental limit of statistical learning.

Imagine a chatbot asked for a recipe for "Lemon Velvet Cake." Suppose the training data contains exactly one blog post with that recipe, and the post accidentally swaps baking powder for baking soda. The model has no other examples to learn from. To it, that single instance is the entire distribution. If the model is calibrated, meaning its probabilities match real-world frequencies, then it must assign some non-zero chance to the wrong ingredient. On rare facts, that chance translates directly into hallucination.

Theoretical work confirms this. A 2023 paper proved that any calibrated language model must hallucinate on "arbitrary" facts that appear only once in training, at a rate tied to the fraction of such singleton facts source. The model cannot distinguish the true fact from a plausible alternative because it has only seen one example. To avoid hallucination, it would have to say "I don't know." But standard training penalizes non-answers just as harshly as wrong answers. So the model learns to guess.

{
  "type": "bar",
  "title": "Hallucination rate vs. training frequency",
  "caption": "Illustrative: Facts appearing only once in training data are almost certain to be hallucinated. Even at higher frequencies, a small error rate remains.",
  "data": [
    {
      "label": "1 occurrence",
      "value": 95
    },
    {
      "label": "10 occurrences",
      "value": 60
    },
    {
      "label": "100 occurrences",
      "value": 20
    },
    {
      "label": "1000 occurrences",
      "value": 5
    }
  ]
}

This also explains why models struggle with recent events, niche topics, or personalized questions like "What did I eat for lunch yesterday?" The training data simply doesn't contain enough signal. The model falls back on patterns from similar contexts, and those patterns may not match reality.

How does the way models are trained make them more likely to confabulate?

During training, the model always sees the correct previous words when predicting the next one. This is called teacher forcing. At inference time, the model must use its own predictions as context. That mismatch is called exposure bias, and it's a major source of hallucination cascades.

Picture a student driver who has only ever ridden with a perfect instructor. The instructor always corrects the wheel before the car drifts. The student never experiences a small mistake and never learns to recover from it. On the real road, a tiny wobble can spiral into a full lane departure because the student has no recovery skill.

For a language model, the first token it generates might be slightly off. Maybe it picks a less common word that still fits. Now that word becomes part of the context for the next prediction. The model has never seen a context that includes its own slightly-off choice during training. So it drifts further. A few tokens later, the output is fluent but completely unmoored from the original intent source.

graph TD
  A["Teacher forcing (training)"] --> B["Sees correct previous tokens"]
  B --> C["Predicts next token well"]
  D["Autoregressive (inference)"] --> E["Sees own generated tokens"]
  E --> F["Small initial error"]
  F --> G["Errors compound over time"]

This is why a chatbot asked to draft a professional email might start with a slightly too casual greeting, then slide into an entirely wrong tone. The first token choice triggers a cascade. The model isn't "deciding" to be unprofessional. It's following its own prior output into a region of text space where the training signal was weak.

Longer generations and higher sampling temperatures both amplify exposure bias. More randomness means more early deviations, which compound. That's one reason why creative writing tasks produce more hallucinations than short factual queries.

Why does the model sound so confident even when it's wrong?

The model's internal state often signals uncertainty. But nothing in its training rewards expressing that uncertainty. Standard benchmarks score answers as right or wrong. A hedged "I think maybe it's X" is treated the same as a wrong answer if X is false. So the model learns to hide its doubt behind a confident surface.

You can see this when you ask a chatbot to summarize a long article that exceeds its context window. The model can't see the middle sections. Instead of saying "I don't have that part," it will often invent plausible-sounding details that fit the surrounding context. It's not trying to deceive you. It's producing the most probable continuation given the incomplete information, and the most probable continuation is a complete, fluent summary.

Researchers have developed ways to measure the uncertainty the model is hiding. One method, semantic entropy, samples multiple answers to the same question and clusters them by meaning. If the model gives semantically diverse answers, its uncertainty is high. High semantic entropy strongly correlates with hallucinated content source. The model "knows" it's uncertain, but it doesn't tell you.

Some newer techniques, like DoLa, exploit the fact that factual knowledge tends to concentrate in specific transformer layers. By contrasting the outputs of deeper and shallower layers during decoding, they can nudge the model toward more truthful tokens without retraining source. These methods work because they tap into the model's own internal signals of factuality, signals that the standard generation process ignores.

Is hallucination a bug or a feature?

It's neither. It's an emergent property of the same mechanism that makes models work. The model's one job is to produce text that is a plausible continuation of the prompt. When plausibility aligns with truth, you get a correct answer. When it doesn't, you get a hallucination. The model isn't switching modes. It's the same prediction engine operating in different regions of the data distribution.

Think back to Part 1: the model predicts the next word. That's it. Everything else, reasoning, creativity, factual recall, is a byproduct of that single trick performed at scale. Hallucination is just the byproduct showing its seams. You can't have the fluent, helpful assistant without also having the occasional confident confabulation, because both come from the same underlying process.

This perspective shifts the engineering challenge. You don't "fix" hallucination like a bug. You design systems around it: retrieval-augmented generation to ground answers in external documents, uncertainty estimators to flag risky outputs, and training incentives that reward appropriate abstention. The model will always be a prediction engine. The goal is to align the contexts where it predicts with the contexts where prediction equals truth.

Mechanism	What it does	Why it causes hallucinations
Maximum likelihood training	Optimizes for next-token accuracy on training text	No truth signal; plausible falsehoods get high probability
Rare facts (singletons)	Facts seen only once in training	Model cannot statistically distinguish truth from alternatives
Teacher forcing	Trains on perfect prefixes only	Model never learns to recover from its own errors; error cascades
Unidirectional attention	Processes left-to-right	Model commits early; cannot revise based on later context
Benchmark incentives	Reward any answer over "I don't know"	Model learns to guess confidently even when uncertain

Frequently Asked Questions

Q: Can prompt engineering eliminate hallucinations?
Good prompts can reduce prompt-induced hallucinations by making the task clearer and constraining the output format. But they cannot fix intrinsic hallucinations caused by missing training data or exposure bias. The model's underlying distribution remains unchanged. Prompt engineering is a mitigation, not a cure.

Q: Why do models hallucinate more on long outputs?
Longer outputs give exposure bias more room to compound. Each token is conditioned on the model's own previous tokens, which may already contain small errors. Over many steps, the context drifts further from the training distribution, increasing the chance of fluent but incorrect text.

Q: Does retrieval-augmented generation (RAG) stop hallucinations?
RAG grounds the model in external documents, which greatly reduces hallucinations caused by outdated or missing training data. But the model can still ignore the retrieved context or confabulate details not present in the documents. RAG reduces the problem; it doesn't eliminate it.

Q: Can a model ever "know" it's hallucinating?
Internally, the model often carries signals of uncertainty, as shown by semantic entropy probes. But it has no built-in mechanism to act on those signals unless explicitly trained to do so. Without special training or decoding, it will produce the most probable token even when internally uncertain.

Q: Will future models stop hallucinating?
As training data and architectures improve, hallucination rates on common facts will drop. But the statistical limits around rare facts and the inherent tension between fluency and truth mean some level of hallucination is likely permanent. The goal is to make it manageable, not to eliminate it entirely.

Test yourself

You're building a customer support chatbot that answers questions about your company's internal policies. The model occasionally invents plausible but incorrect policy details when the question is slightly ambiguous. You've already improved the prompts. What's one architectural change you could make that directly addresses the root cause of these hallucinations, and why would it help?

Answer: Add a retrieval step that fetches the exact policy document paragraphs relevant to the question and prepends them to the prompt (RAG). This works because the hallucinations are likely caused by the model falling back on generic patterns from its training data when the prompt is ambiguous. By injecting the ground-truth policy text into the context window, you shift the model's distribution: the most probable continuation given that specific context is now a faithful paraphrase or quote, not a confabulation. This doesn't fix the model's internal tendency to guess, but it changes the input so that guessing isn't necessary. The model can simply attend to the provided text, much like it does when summarizing a visible article.

If this kind of breakdown, how real systems actually work under the hood, is what you're after, subscribe to Internals Decoded at internalsdecoded.com. Next time, we'll look at how models can be steered to be more helpful and honest without breaking their core prediction engine.

Sources

Originally published at Internals Decoded. AI internals, explained conversationally.

Context Windows: The Model's Working Memory

Internals Decoded — Thu, 27 Aug 2026 23:00:12 +0000

A context window is the maximum number of tokens a language model can process in one go. It includes the system prompt, conversation history, and the model's own output. Inside the model, self-attention and a key-value cache enforce this limit, acting as the model's working memory. When the window fills up, older information falls out, and the model can no longer use it.

But here is the twist: the model's performance drops even when the window is not full. If you bury a crucial detail in the middle of a long prompt, the model often ignores it. And in a long chat, the assistant might start hallucinating long before you hit the advertised token limit.

In the last episode, we saw how training and inference are separate phases. Now we will look at the inference-time machinery that decides how much of your conversation the model can actually keep in mind.

What exactly is a context window?

Imagine you are reading a long recipe on a small phone screen. You can only see a few lines at a time. To follow the recipe, you scroll up and down, but you cannot see the whole thing at once. The visible area is your context window.

A language model works the same way. It reads text in chunks called tokens, not words. A token is a small piece of text, often a word fragment. The model can only “see” the tokens that fit inside its context window. If the recipe is 10,000 tokens and the window is 4,000, the model will only read the first 4,000 tokens (or the last 4,000 if you truncate from the beginning). It will miss the baking temperature at the end.

{
  "type": "stat",
  "title": "Context Window at a Glance",
  "caption": "A typical LLM context window compared to human reading. Illustrative values based on common API limits and average reading speeds.",
  "stats": [
    {
      "value": "128k",
      "label": "Max tokens (GPT-4 Turbo)"
    },
    {
      "value": "~100k",
      "label": "Words that fits"
    },
    {
      "value": "~1.3",
      "label": "Tokens per word"
    },
    {
      "value": "~5 min",
      "label": "Human reading time"
    }
  ]
}

Every message you send and every word the model generates consumes tokens from the same finite window. When you ask a chatbot for a recipe, the prompt plus the model's reply might use 500 tokens. If you then ask “what was the first ingredient?”, the model needs the earlier conversation to answer. But if the total token count exceeds the window, the earliest messages get pushed out, and the model literally cannot see them anymore. This is why long chats eventually lose track of what was said at the beginning. OpenAI tokenizer

How does the model “remember” everything within the window?

Think of the model as a reader who takes notes on a scratchpad. Instead of re-reading the whole book each time they write a new sentence, they jot down key points about every previous sentence. That scratchpad is the key-value cache, or KV cache.

When you first send a prompt, the model does a full read, called the prefill phase. It processes every token and stores two vectors for each token at every layer: a key and a value. These vectors capture what the token means and how it relates to others. Once the prefill is done, the model has a complete set of notes for the entire input.

When it generates the next word, it only needs to look at its cached notes and the new word. It does not re-process the whole history. This is why generation after the first token is fast. The KV cache grows with each new token, and it lives in GPU (graphics processing unit) memory. For a model with many layers and heads, the cache can easily become larger than the model weights themselves. Attention Is All You Need

A simplified view of the process looks like this:

# Pseudocode for autoregressive decoding with KV cache (simplified)
tokens = tokenize(prompt)
logits, kv_cache = model.forward_prefill(tokens)
next_token = sample_from_logits(logits[-1])
generated = [next_token]

for _ in range(max_new_tokens - 1):
    logits, kv_cache = model.forward_decode(next_token, kv_cache)
    next_token = sample_from_logits(logits)
    generated.append(next_token)

text = detokenize(generated)

Why does the context window have a hard limit?

The attention mechanism is the reason. Attention lets every token look at every other token to decide what is important. If you have 4,000 tokens, the model computes 16 million pairwise comparisons per layer. Double the window to 8,000 tokens, and you quadruple the comparisons. This quadratic growth quickly becomes too expensive in time and memory.

Even with the KV cache, each new token still attends to all previous tokens. So the cost per generated token is linear in the sequence length, and the memory for the cache grows linearly too. A 128,000-token context can require hundreds of gigabytes of GPU memory just for the cache. That is why every model has a maximum sequence length baked into its architecture. Attention Is All You Need

{
  "type": "comparison",
  "title": "Attention Cost: Short vs. Long Context",
  "caption": "Computational cost of the attention mechanism scales quadratically with sequence length. Values are illustrative for a single layer.",
  "before": {
    "label": "4k tokens",
    "points": [
      "16 million comparisons",
      "~0.5 GB memory",
      "Fast prefill"
    ]
  },
  "after": {
    "label": "128k tokens",
    "points": [
      "16 billion comparisons",
      "~64 GB memory",
      "Slow prefill"
    ]
  }
}

In the recipe analogy, if you had to compare every ingredient with every other ingredient to understand the recipe, a longer recipe would become impossibly slow. The context window is the model's way of saying “I can only handle this many comparisons at once.”

Why do long conversations get weird?

Two phenomena cause trouble: the “lost in the middle” effect and the mismatch between advertised and training context lengths.

Models pay the most attention to the beginning and the end of the context. If you put a crucial instruction in the middle of a long prompt, the model often ignores it. In a chat, a follow-up question that relies on something said 20 turns ago (now sitting in the middle of the window) may get a wrong answer. The model simply does not give that middle region the same weight. Lost in the Middle

{
  "type": "donut",
  "title": "Where Attention Goes",
  "caption": "Models often over attend to the beginning and end of the context, causing the 'lost in the middle' effect. Illustrative distribution based on research findings.",
  "data": [
    {
      "label": "Beginning (primacy bias)",
      "value": 40
    },
    {
      "label": "Middle (lost information)",
      "value": 20
    },
    {
      "label": "End (recency bias)",
      "value": 40
    }
  ]
}

The second issue is that the model's training context length is often smaller than the API (application programming interface)'s maximum. Llama 3, for example, was trained on 8,192-token sequences. When you push it to 32,000 tokens, the positional encodings that tell the model the order of words become unreliable. The model was never taught to handle positions that far out, so its predictions degrade. This happens even if the GPU has plenty of memory. Llama 3 technical report

In stateful multi-turn apps that reuse the KV cache across turns, the cache grows monotonically. Once it exceeds the training context length, quality drops sharply. The assistant might start repeating itself, contradicting earlier statements, or inventing facts. This is why long conversations get weird long before you hit the advertised token limit.

How do systems try to extend or manage the context window?

The simplest method is truncation: keep only the most recent N tokens and drop the rest. This is like scrolling the phone screen so you always see the last few lines. It works, but you lose all earlier context.

A smarter approach is retrieval-augmented generation (RAG). Instead of stuffing the entire knowledge base into the window, you search for relevant documents and put only those into the prompt. This keeps the window small and focused. RAG paper

Some systems try to compress the KV cache itself. They use learned modules that distill a long cache into a fixed-size summary. This is like writing a one-paragraph summary of the recipe instead of keeping every line. However, these methods must be careful with positional encodings. If you delete tokens from the middle of the cache, you break the model's sense of order. Many compression strategies fail because they scramble the relative positions that models like Llama 3 rely on. KV cache compression survey

The most robust strategy today is often the simplest: keep a contiguous block of the most recent conversation and either summarize or discard the rest. This preserves positional integrity, even if it throws away more tokens.

Quick Reference

Property	Value
Typical token-to-word ratio (English)	~1.3 tokens per word
Common context lengths	4k (GPT-3.5), 128k (GPT-4 Turbo), 1M (Gemini 1.5 Pro)
KV cache memory per token (FP16)	2 × layers × heads × head_dim × 2 bytes
Attention complexity	O(n²) for full attention
“Lost in the middle” effect	Accuracy drops when relevant info is in the middle of the context

Frequently Asked Questions

Q: Does a larger context window always mean better performance?
No. Models may not be trained to use very long contexts effectively. Attention can get diluted, and retrieval quality often drops as the window grows.

Q: Why does my assistant forget the beginning of the conversation even though it claims 128k context?
The app probably truncates the history to save cost or latency. Or the model's effective context is much smaller than the advertised number because of training limits.

Q: How does RAG compare to just using a huge context window?
RAG fetches only the relevant information, reducing noise and compute. A huge window forces the model to sift through everything, which can hurt accuracy and increase latency.

Q: What is the difference between architectural context limit and effective context limit?
The architectural limit is the maximum number of tokens the model can physically process. The effective limit is where the quality remains acceptable. The effective limit is often much lower.

Q: Can I train a model to have an infinite context window?
Not with standard attention. You would need architectural changes like linear attention or external memory. These are still active research areas.

Test yourself

You are debugging a customer support bot that uses a long conversation history. After 50 turns, the bot starts giving irrelevant answers. What is likely happening, and how would you diagnose it?

Answer: The accumulated KV cache has likely grown beyond the model's architectural context limit. Even if the API says 128k tokens, the model may have been trained on only 8k. Once the cache exceeds that, positional encodings misalign and attention degrades. You can check the total token count of the conversation. Try truncating to the last 20 turns and see if quality improves. Also, look for “lost in the middle”: important context from earlier turns may now sit in the middle of the window and be ignored. Moving that context to the very beginning or end of the prompt can help. If the problem persists, implement a summarization step that compresses older turns into a short gist before the cache grows too large.

If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.

Sources

Originally published at Internals Decoded. AI internals, explained conversationally.

Training vs Inference: Why Building Costs Millions and Asking Costs Cents

Internals Decoded — Tue, 25 Aug 2026 14:11:23 +0000

In the last episode, we saw how embeddings turn words into numbers that capture meaning. That is the model’s internal language. But the model itself had to learn those embeddings, along with everything else, during a phase called training. The moment you ask it for a recipe, it’s using a completely different phase: inference. These two phases are often confused, but they are as different as building a car and driving it.

Here is the twist: for a popular chatbot, the total cost of inference over its lifetime often dwarfs the training cost. That changes how you think about what is “expensive.” The million-dollar training run gets the headlines. The pennies per query add up to millions every year, silently.

What actually happens when you train a large language model?

Training is a massive optimization problem. The model sees billions of example sentences, tries to predict the next word, measures how wrong it is, and adjusts its billions of parameters to get better. Think of it like a student reading the entire internet. After every sentence, the student takes a pop quiz: “What word comes next?” They check the answer, note every mistake, and update their mental rules. That loop runs for weeks on thousands of GPUs.

During training, the model processes a batch of token sequences. It runs a forward pass through all its layers to produce a prediction for each position. Then it computes a loss, usually cross-entropy, between its predictions and the real next tokens. The backward pass calculates how every parameter contributed to the error. Finally, an optimizer like Adam updates each parameter to reduce the loss next time. This forward-backward-optimizer cycle is one training step.

The math behind the cost is straightforward. For a transformer with P parameters, one forward pass costs about 2P floating-point operations (FLOPs) per token. Adding the backward pass and optimizer update brings the total to roughly 6P FLOPs per token. GPT-3 has 175 billion parameters and was trained on about 300 billion tokens. That works out to around 3.15×10²³ FLOPs, or 3,640 petaflop-days. No single GPU (graphics processing unit) can run that in a human lifetime. Training must be distributed across clusters of hundreds or thousands of accelerators.

Distributed training uses several tricks. Data parallelism gives each GPU a full copy of the model and a different slice of data. After each step, GPUs average their gradients. Tensor parallelism splits individual weight matrices across GPUs, so each device only stores a shard. Pipeline parallelism divides layers into stages, passing micro-batches through like an assembly line. Systems like Megatron-LM and DeepSpeed ZeRO combine all three to fit enormous models into limited memory and keep GPUs busy. The cluster behaves like one giant computer, bound by network bandwidth as much as by raw FLOPs.

So the model you chat with was not born knowing about chocolate cake. It read millions of recipes, forum posts, and cookbooks during training. It learned patterns like “butter, sugar, flour” appearing together. That education cost millions of dollars and took months.

What happens when you ask a model a question?

Inference is the model applying what it learned to a new input. It processes your prompt in a single forward pass, then generates one token at a time. The key trick that makes this fast is a KV (key-value) cache. It avoids recomputing attention for tokens it has already seen.

You open a chatbot and type, “Give me a recipe for chocolate cake.” The system tokenizes your text into a sequence of token IDs. Then it enters the prefill phase. All input tokens go through the model at once. For each token at each layer, the model computes key and value vectors and stashes them in the KV cache. At the end, it produces a logit for the first new token. This prefill step is compute-heavy but happens only once per prompt.

Now the decode phase begins. The model takes that first generated token, embeds it, and runs a forward pass that reads the stored keys and values from the cache. It only computes new key, value, and query vectors for the current token. It attends over the entire prefix without recomputing anything. The output is a logit for the next token. The process repeats, “Sure!”, “Here”, “is”, “a”, “simple”, “recipe”, until a stop token appears.

graph TD
A["Input prompt"] --> B["Tokenize"]
B --> C["Prefill: process all tokens, fill KV cache"]
C --> D["Decode: generate one token"]
D --> E{"More tokens?"}
E -->|Yes| D
E -->|No| F["Output complete"]

Each decode step still reads the entire model’s weights from GPU memory. But the math per byte of weight data is tiny. This makes the decode phase memory-bandwidth-bound. The GPU’s compute units spend most of their time waiting for data from HBM. That is why inference hardware prizes memory bandwidth over peak FLOPs.

The KV cache grows with sequence length. For a model with L layers, H attention heads, and head dimension d, each token adds 2 × L × H × d numbers to memory. For long conversations, the cache can rival the model weights in size. Managing that memory is a central challenge of serving.

Why does training cost millions while inference costs cents?

Training requires thousands of GPUs running for weeks. Each token processed triggers a full forward and backward pass plus optimizer state updates. The total FLOPs are enormous. Electricity, hardware depreciation, and engineering time add up. Estimates for GPT-3’s training run range from $4 million to $12 million. Larger models like GPT-4 likely cost over $100 million.

Inference, on the other hand, runs a forward-only pass. The cost per token is roughly one-third of a training token’s FLOPs. A single query might involve a few hundred input tokens and a few hundred output tokens. At a typical price of $0.001 per 1,000 tokens, that query costs a fraction of a cent. A single GPU can serve dozens of users concurrently.

{
  "type": "bar",
  "title": "FLOPs per token (relative to model parameters P)",
  "caption": "Training requires forward and backward passes, tripling the FLOPs compared to inference.",
  "data": [
    {
      "label": "Training",
      "value": 6
    },
    {
      "label": "Inference",
      "value": 2
    }
  ]
}

But the cents add up. A service handling 100 million queries per day at $0.002 per request spends about $73 million per year on inference. Over a model’s lifetime, inference often dominates total cost. The training bill is a one-time capital expense. Inference is an ongoing operational expense that scales with every new user.

{
  "type": "stat",
  "title": "Lifetime cost comparison",
  "caption": "Training cost is illustrative; inference cost based on 100M queries per day at $0.002 each.",
  "stats": [
    {
      "value": "~$10M",
      "label": "Training cost (one time)"
    },
    {
      "value": "$73M",
      "label": "Inference cost per year"
    }
  ]
}

This explains why so much engineering effort in 2024-2026 has focused on inference optimization. Techniques like quantization, speculative decoding, and PagedAttention squeeze more tokens per second out of the same hardware. They attack the memory bandwidth and KV cache bottlenecks directly.

Why is inference more about memory bandwidth than compute?

During decode, the model reads every one of its billions of weights from GPU memory for each token it generates. The arithmetic intensity, FLOPs per byte of data moved, is low. A modern GPU like an H100 can perform over a thousand trillion FLOPs per second, but its memory bandwidth is “only” 3.35 terabytes per second. The math finishes quickly, and the GPU stalls waiting for the next chunk of weights.

Think of a librarian fetching books from miles of shelves. The bottleneck is how fast they can walk, not how fast they can read the title. For inference, faster memory (higher bandwidth HBM) directly translates to more tokens per second. That is why the H200, with its larger and faster memory, outperforms the H100 on decode-heavy workloads even though peak FLOPs are similar.

Prefill is different. It processes many tokens in parallel, doing large matrix multiplications. There, compute throughput matters. But for chat, where output tokens outnumber input tokens, decode dominates. Memory bandwidth is king.

How do serving systems handle multiple users at once?

A naive approach would process one request at a time. The GPU would sit idle during the memory-bound decode steps. Modern serving engines use continuous batching. They group requests that arrive at different times into a single batch. When one request finishes, a new one can join immediately. This keeps the GPU fed with enough work to hide memory latency.

They also manage KV caches intelligently. PagedAttention, introduced by the vLLM project, treats the KV cache like virtual memory. It allocates cache in blocks and maps them to sequences, avoiding fragmentation and allowing memory sharing when prompts share a prefix. This increases the number of concurrent users a single GPU can serve.

These systems turn a handful of GPUs into a service that handles thousands of simultaneous conversations. The same model that required a supercomputer to train now runs on a commodity server, answering recipe requests for pennies.

Quick Reference

Property	Training	Inference
Passes	Forward + backward + optimizer	Forward only
FLOPs per token	~6P	~2P
Dominant constraint	Compute FLOPs, memory	Memory bandwidth, KV cache memory
Typical batch size	Large (hundreds to thousands of sequences)	Small to medium, dynamically changing
Parallelism	Data, tensor, pipeline, ZeRO	Request-level batching, tensor parallel for huge models
Time horizon	Weeks to months per run	Continuous 24/7
Cost profile	Large upfront capex	Per-token marginal cost, accumulates over lifetime
GPU utilization goal	Maximize throughput	Maximize throughput under latency SLOs

Frequently Asked Questions

Q: Does fine-tuning count as training or inference?
Fine-tuning is training, just on a smaller scale. It runs forward and backward passes to update the model’s weights, using a fraction of the original compute. The cost is far lower than pretraining but still uses training infrastructure.

Q: Why can’t we use the same hardware for both training and inference?
You can, but it is usually wasteful. Training needs high-bandwidth interconnects between thousands of GPUs and favors peak FLOPs. Inference benefits from high memory bandwidth and smaller, cheaper clusters. Dedicated inference hardware like the L40S or H200 is optimized differently.

Q: How does the KV cache affect latency?
The KV cache grows linearly with context length. For very long conversations, reading and updating the cache can become a bottleneck. If the cache exceeds GPU memory, the system must spill to CPU (central processing unit) or disk, causing latency spikes. Efficient cache management is critical for long-context serving.

Q: What is the biggest cost driver in a production LLM service?
For most chat applications, the decode phase dominates. Output tokens are generated one by one, each requiring a full model weight read. Memory bandwidth and the number of concurrent users determine how many GPUs you need. Reducing per-token cost through quantization and batching has the largest impact on total spend.

Q: Can inference be done on CPUs?
Yes, but it is slow for large models. CPUs have much lower memory bandwidth than GPUs. A model that generates 20 tokens per second on a GPU might manage 1-2 tokens per second on a high-end CPU. For latency-sensitive chat, GPUs are necessary. For batch processing where latency is less critical, CPUs can be cost-effective.

Test yourself

You are building a customer support chatbot that handles 10,000 conversations per day. Each conversation averages 500 input tokens and 200 output tokens. Your model provider charges $0.001 per 1,000 tokens for inference. What is the daily compute cost? How would you reduce it if the service grew 100x?

Answer: Total tokens per conversation: 500 + 200 = 700 tokens. Daily tokens: 10,000 × 700 = 7,000,000 tokens. Cost: 7,000 × $0.001 = $7.00 per day. At 100x growth, that becomes $700 per day, or about $255,000 per year. To reduce costs, you could switch to a quantized model (often half the cost with minimal quality loss), implement semantic caching so identical or similar questions reuse previous answers, or batch requests during off-peak hours to get volume discounts. You might also fine-tune a smaller model on your support data so it can run on cheaper hardware, or use speculative decoding to generate multiple tokens per step, cutting the number of forward passes.

If you want this kind of breakdown every week, how real AI systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.

Next time, we will look at how models are fine-tuned to follow instructions. That is the step that turns a raw text predictor into a helpful assistant.

Sources

Originally published at Internals Decoded. AI internals, explained conversationally.

Embeddings: Meaning as Numbers

Internals Decoded — Sun, 23 Aug 2026 13:51:14 +0000

When you ask a chatbot for a pasta recipe, it doesn’t understand “pasta” the way you do. It sees a list of numbers. Those numbers are not random. They are arranged so that “pasta” sits close to “spaghetti” and far from “car.” This is an embedding. It turns meaning into coordinates that a machine can measure, compare, and search. Every time a chatbot finds a relevant answer, recommends a product, or remembers a fact from earlier in the conversation, embeddings are doing the heavy lifting behind the scenes.

Here is the surprising part. The machine never learns what “pasta” means. It only learns which words appear in the same kinds of sentences. That statistical shadow turns out to be so rich that it captures everything from synonyms to analogies. The rest of this article unpacks how that happens, step by step, using the same chatbot interactions you already know from this series.

What exactly is an embedding?

An embedding is a list of numbers that represents something discrete, a word, a token, a user ID, a product. Each number is a coordinate in a high-dimensional space. If you pick the right coordinates, similar things end up near each other. That is the entire idea.

Think of a map. A city’s latitude and longitude don’t tell you its name or history. But if you know that Paris is at (48.9, 2.3) and Lyon is at (45.8, 4.8), you can measure the distance and see they are both in France. Embeddings work the same way, except they use hundreds of dimensions instead of two. Every dimension captures some latent feature of the input, learned from data. The model never labels those features. It just arranges points so that words that behave similarly in text end up with similar coordinates.

In a chatbot, every token from Part 2 gets its own embedding vector. The model then uses those vectors as the starting point for everything else, including the attention mechanism from Part 3. The quality of the embeddings directly determines how well the model can tell that “I need a quick dinner idea” and “fast evening meal suggestions” mean roughly the same thing.

How does a chatbot turn words into numbers?

You already know that the chatbot breaks your message into tokens. For the query “give me a pasta recipe,” the tokenizer might produce token IDs like [123, 45, 678, 901]. Those integers are just labels. They carry no meaning on their own.

The model has a large table called an embedding matrix. It has one row for every token in the vocabulary. Each row is a vector of, say, 768 numbers. When the model sees token ID 678, it looks up row 678 and pulls out that vector. This lookup is the embedding layer. It turns a sequence of token IDs into a sequence of dense vectors that the rest of the network can process.

graph TD
A["Input text 'pasta recipe'"] --> B["Tokenizer produces IDs"]
B --> C["Embedding matrix lookup"]
C --> D["Vector for token 'pasta'"]
C --> E["Vector for token 'recipe'"]

This lookup is fast and differentiable. During training, the model adjusts the numbers in those rows so that the vectors become useful for the task at hand. If the model is trained to predict the next word, then words that lead to similar next-word predictions will gradually drift toward each other in the vector space. The embedding layer itself has no built-in notion of meaning. It is just a giant spreadsheet of numbers that gets updated by backpropagation.

The same mechanism applies to any discrete input. When a recommendation system sees your user ID, it fetches a vector that represents your preferences. When a search engine indexes a document, it stores an embedding vector for the whole document. The table lookup is the universal first step for turning symbols into numbers a neural network can digest.

Why do similar words end up with similar vectors?

The key insight is the distributional hypothesis. Words that appear in similar contexts tend to have similar meanings. If you see the word “pasta” surrounded by “sauce,” “boil,” “dinner,” and “recipe,” and you see “spaghetti” in the exact same kinds of sentences, then a model that learns from those contexts will place “pasta” and “spaghetti” close together. It doesn’t know what either word means. It only knows they are interchangeable in many sentences.

Early embedding methods made this explicit. They would scan a huge corpus and count how often every pair of words appeared near each other. That produced a giant co-occurrence matrix. Then they would squash that matrix down to a small number of dimensions using a technique like singular value decomposition. The result was a compact vector for each word that preserved the most important co-occurrence patterns. This is like taking a huge spreadsheet of word relationships and compressing it into a few columns that capture the gist.

Modern models do something similar, but they learn the vectors on the fly while training a neural network. They don’t count everything first. Instead, they look at a small window of words, try to predict a target word from its neighbors (or vice versa), and adjust the vectors to get better at that prediction. Over millions of examples, the vectors settle into a geometry where words that predict the same contexts cluster together.

This is why embeddings can solve analogies. The classic example is “king minus man plus woman equals queen.” The vectors don’t store royal titles. They store the directions that separate gender and royalty, learned from thousands of sentences where these words appear in parallel roles.

How does the model learn these vectors?

The most famous example is Word2Vec’s skip-gram model with negative sampling. The idea is simple. Take a sentence like “I need a quick pasta recipe.” Slide a window over it. For each center word, like “pasta,” pick a nearby context word, like “recipe.” The model’s job is to decide whether this pair is real or fake.

The model computes a dot product between the embedding of “pasta” and the embedding of “recipe.” A large positive dot product means the model thinks they belong together. A negative or small dot product means they don’t. The model is also shown fake pairs, like “pasta” and “elephant,” sampled at random. It must learn to give high scores to real pairs and low scores to fake ones.

{
  "type": "bar",
  "title": "Dot product similarity between word embeddings",
  "caption": "Illustrative similarity scores. High scores indicate words that often appear together.",
  "data": [
    {
      "label": "pasta - recipe",
      "value": 0.8
    },
    {
      "label": "pasta - sauce",
      "value": 0.75
    },
    {
      "label": "pasta - car",
      "value": 0.1
    },
    {
      "label": "pasta - quantum",
      "value": -0.2
    }
  ]
}

Over time, the embeddings shift so that words that often appear together get vectors with large dot products. Words that never appear together get vectors that point in different directions. Because the model sees millions of real and fake pairs, it ends up encoding subtle semantic relationships. “Pasta” and “spaghetti” both appear with “recipe,” so they both get pushed toward the same region of the space.

Researchers later proved that this process is equivalent to factorizing a matrix of pointwise mutual information (PMI) between words and contexts. PMI measures how much more often two words co-occur than you would expect by chance. The model is essentially learning a low-dimensional approximation of that PMI matrix. That is why the numbers capture meaning. They are a compressed representation of statistical association.

In practice, the model never builds the full PMI matrix. It learns the vectors directly from the data stream, which scales to billions of words. The embedding dimension, typically a few hundred, is a hyperparameter that controls how much information can be packed into each vector. More dimensions can capture finer distinctions, but they also cost more memory and computation.

How are embeddings used beyond single words?

A chatbot needs to understand whole sentences, not just individual words. After the token embeddings are looked up, the transformer layers from Part 3 mix them together using attention. The final output is a sequence of vectors, one per token, each now carrying information about the entire sentence.

To get a single vector for the whole sentence, a common trick is to take the vector corresponding to a special token, like [CLS], or to average all token vectors. This sentence embedding can then be compared to other sentence embeddings using cosine similarity. If two sentences have a high cosine similarity, the model considers them semantically close.

This is how a chatbot knows that “how do I make pasta” and “pasta cooking instructions” are asking the same thing. The query is turned into a sentence embedding, and the chatbot compares it to embeddings of candidate answers or documents. The one with the highest similarity is returned.

Modern sentence embedding models are often trained with contrastive learning. They take a sentence, create two slightly different versions (like by dropping words or using a paraphrase), and force the model to make those two versions have nearly identical embeddings. At the same time, they push embeddings of unrelated sentences apart. This makes the embedding space more uniform and reliable for similarity comparisons.

This same idea powers recommendation systems. A user’s interaction history can be averaged into a user embedding. Items get their own embeddings. The dot product between user and item embeddings predicts how much the user will like the item. Behind every “you might also like” is a nearest-neighbor search in embedding space.

What does this mean for search, recommendations, and memory?

When you type a query into a chatbot, it doesn’t do a keyword match. It converts your query into an embedding and finds the stored passages whose embeddings are closest. This is semantic search. It works even when the words don’t overlap at all. “Inexpensive lodging near the beach” and “budget hotel by the sea” will map to nearby points, so the system retrieves the same results.

This is also how a chatbot can “remember” facts from earlier in a conversation. Some architectures store the conversation history as a set of embeddings and retrieve relevant parts when you ask a follow-up question. The model doesn’t have a photographic memory. It has a vector database that finds sentences with similar meaning to your new query.

Recommendation systems use the same principle. Every song, movie, or product gets an embedding learned from user behavior. The system finds items whose embeddings are close to yours. The math is the same as word similarity, just applied to different kinds of entities.

The takeaway is that embeddings are the universal adapter. They turn messy, discrete things, words, sentences, users, items, into a clean, continuous geometry where distance equals similarity. Every modern AI system that deals with language or personalization relies on this trick.

Quick Reference

Concept	Plain English
Embedding vector	A list of numbers that represents a word, token, or item
Embedding dimension	The length of that list (often 300-4096)
Embedding matrix	A table with one row per vocabulary item, each row a vector
Distributional hypothesis	Words that appear in similar contexts have similar meanings
Co-occurrence	How often two words appear near each other in text
PMI (pointwise mutual information)	A measure of how much more often words co-occur than chance
Negative sampling	Training by contrasting real word pairs with random fake pairs
Cosine similarity	A measure of angle between vectors, used to compare embeddings
Sentence embedding	A single vector that represents a whole sentence
Contrastive learning	Training to pull similar pairs together and push dissimilar pairs apart

Frequently Asked Questions

Q: Do embeddings really capture meaning, or just word co-occurrence?

They capture co-occurrence patterns that happen to align with meaning. The model never accesses definitions or real-world knowledge. But for many practical tasks, the statistical shadow of meaning is enough. That is why “pasta” and “spaghetti” end up close, even though the model has never tasted either.

Q: Why are embeddings high-dimensional? Couldn’t we use just a few numbers?

A few dimensions can’t capture the many different ways words can be similar. “King” and “queen” are similar in royalty but differ in gender. “King” and “ruler” are similar in role but differ in formality. High-dimensional spaces give the model enough room to represent many independent axes of variation simultaneously.

Q: How do I choose the right embedding dimension?

It’s a tradeoff. Larger dimensions can capture more nuance but require more memory and slow down computation. For word embeddings, 300 is a classic sweet spot. For sentence embeddings from transformers, 768 or 1024 is common. Start with what the pretrained model provides and only reduce if you have tight latency constraints.

Q: Can I use embeddings for languages other than English?

Yes. The same principles apply to any language. Multilingual models train on many languages at once and learn a shared embedding space where “pasta” in English and “pâtes” in French end up close if they appear in similar translated contexts. This enables cross-lingual search without explicit translation.

Q: Are embeddings the same as the hidden states inside a transformer?

Not exactly. The token embeddings are the input to the transformer. The hidden states are the outputs after attention has mixed the token embeddings with context. Both are vectors, but hidden states are context-dependent: the vector for “bank” will be different in “river bank” versus “bank account.” Embeddings are context-independent lookup values.

Test yourself

You are building a chatbot that helps users find recipes. A user types “I want something warm and comforting for dinner.” The chatbot retrieves a recipe for chicken soup, which is correct. But when the user types “I need a cozy meal for tonight,” the chatbot retrieves a salad recipe. You suspect the embedding model is not capturing the similarity between “warm and comforting” and “cozy.” How would you diagnose and fix this?

Answer: First, check the cosine similarity between the sentence embeddings of the two queries. If it is low, the model sees them as unrelated. Next, examine the token embeddings for “warm,” “comforting,” and “cozy.” If “cozy” is far from the others, the pretraining data likely lacked examples where “cozy” appeared in food contexts. To fix this, you can fine-tune the sentence embedding model on a small dataset of paraphrased recipe queries, using contrastive learning to pull “cozy meal” and “warm comforting dinner” together. Alternatively, you can augment the retrieval index with synonyms or use a cross-encoder reranker that compares the query and candidate directly, which is more accurate but slower. The root cause is that the generic embedding space doesn’t specialize in your domain’s phrasing, and fine-tuning or reranking bridges that gap.

If you want this kind of breakdown every week, how real AI systems actually work under the hood, from tokens to attention to embeddings and beyond, subscribe to Internals Decoded at internalsdecoded.com. The next episode will show how these embeddings are used to build a model’s memory, allowing it to recall facts from its training data without storing every sentence.

Sources

Originally published at Internals Decoded. AI internals, explained conversationally.

Attention, Gently: How Models Decide What Matters

Internals Decoded — Fri, 21 Aug 2026 14:05:03 +0000

In the first two parts of this series, you saw that a large language model’s whole job is to guess the next word, and you learned that it chops your sentences into tiny pieces called tokens. The question still hanging in the air is: how does the model figure out which pieces of the whole conversation are relevant when it’s about to pick the next token? That decision, the mechanism that decides what to pay attention to, is what this article explains, in simple terms, with no math.

It’s not magic. It’s a system of asking questions and listening to answers that runs tens of thousands of micro-decisions every second. When you chat with a bot, this system is the reason it can remember that you mentioned “a vegan lasagna” three paragraphs ago and still pull that detail into the current reply. But here’s the part that surprises most engineers: the colorful attention maps you see in demos, those bright lines connecting words, are not actually what the model uses to decide what matters. Those maps are a by-product, not the driver. The real machinery is a set of learned projections that route information, and they work in ways that visualizations often mislead us about. Let’s take it apart.

How does attention decide which tokens matter?

Attention decides which tokens matter by generating a query that describes what the current position is looking for, and comparing it to keys that describe what every other token offers. The model then uses the comparisons to grab a weighted blend of the actual information, the values, from all the tokens it can see. The result is a new representation for the current token that has “listened” to the rest of the sequence in a very deliberate, learnable way.

Think of a crowded room where everyone is a token. Each person carries three things. They have a question they want answered right now (the query). They also wear a name tag that states what kind of information they can provide (the key). And they hold a piece of paper with the actual facts they know (the value). When a person needs to decide what to say next, say, the word “lasagna”, they walk around the room mentally. For every other person, they check how well that person’s name tag matches their current question. If the tag says “I know about ingredients” and the question is “What else goes with spinach?”, the match is strong. The listener then takes the facts from that person, weighted by how good the match was, and blends them into a collective whisper. That blended whisper is attention’s output: a summary of the most relevant information the entire room could give.

In a real transformer, these questions, tags, and facts are all vectors. A small learned projection takes the same input token and creates three different versions of it, one for each role. The dot product between a query and a key gives a raw similarity score. Passing those scores through a softmax turns them into a set of positive weights that sum to one. Finally, the weights act as mixing coefficients on the value vectors. This whole operation, scaled dot-product attention, is the core engine behind every modern language model. The original paper that introduced it is Attention Is All You Need, and the scaling factor, dividing by the square root of the key dimension, is a practical fix that stops the softmax from becoming overly confident before the model has learned anything.

graph TD
A["Input token"] --> B["Q, K, V projections"]
B --> C["Score = Q dot K^T"]
C --> D["Apply mask if needed"]
D --> E["Softmax"]
E --> F["Weighted sum of V"]
F --> G["Output"]

But a single set of questions, name tags, and answers can only capture one type of relationship at a time. A token might need to focus on grammar, topic consistency, and the immediate next word all at once. That’s why attention almost never works alone.

Why does attention use multiple heads?

Multi-head attention lets the model pay attention to several different kinds of relationships at the same time. Instead of running one attention calculation per layer, the model runs many in parallel, each with its own learned projections for query, key, and value, and then combines their outputs. This gives the model the ability to look at the conversation from different angles simultaneously.

Imagine the same room full of people, but this time there are several small groups of specialists. One group is tuned to listen for grammar patterns, another tracks the topic of the discussion, a third spots contradictions. When a person speaks, they listen to all the groups at once and then mix the advice they receive into a single refined statement. That mixing is exactly what the final linear projection in a multi-head attention block does. Each head operates in a lower-dimensional subspace, and the model can dedicate different subspaces to different relational patterns. In practice, researchers have found individual heads that specialize in copying recent words, in resolving pronouns like “it” to earlier nouns, or in detecting quotation boundaries. That’s why editing just a handful of heads can sometimes fix a specific failure without retraining the whole model. The multi-head design is not an optional add-on; even in small models, a single head is too rigid to handle the variety of connections real language demands.

Now, not all relationships are allowed. During generation, the model must never peek at words it hasn’t written yet. That restriction is enforced by masking.

How does attention handle what the model can and cannot see?

Masking stops attention from looking at tokens that should be invisible for the current task. In autoregressive generation, a causal mask blocks every position from attending to future positions. Without this, the model would cheat by reading the answer before it’s generated, and training would collapse.

Implementing the mask is straightforward. Before the softmax, any score that corresponds to a forbidden connection gets a huge negative value pushed into it, effectively minus infinity, so the softmax probability for that connection becomes zero. The resulting attention pattern is an upper-triangular matrix of zeros and non-zero weights, which means each token can only mix information from itself and tokens that appeared earlier in the sequence. Padding masks work the same way. If a batch of inputs has different lengths, the shorter ones get empty filler tokens, and the mask ensures the model never accidentally blends in those empty slots.

When the chatbot talks to you, its decoder layers use causal self-attention over the conversation history built so far. For encoder-decoder architectures, there’s also cross-attention, where the decoder uses its own queries but grabs keys and values from the encoded representation of the user’s prompt. That cross-attention has no causal restriction on the encoder side, so the decoder can freely look back at the entire input at every generation step. But even with perfect masking, attention still has a blind spot: it cannot, by itself, tell whether “the cat sat” and “the sat cat” are the same.

How does attention know the order of words?

Attention knows the order of words because positional information is explicitly injected into the token representations before they reach any attention layer. Without this injection, a bag-of-words would all look the same to the model, and it would confuse “dog bites man” with “man bites dog.”

The original transformer added a fixed sinusoidal pattern to each token embedding, with different frequencies encoding absolute position. Later models switched to learned position embeddings, which the model can tune during training. More recent designs like rotary position embeddings (RoPE) encode relative position directly into the attention score calculation, allowing the model to better handle sequences longer than any it saw during training. This is important because it means the same question-and-answer pattern can work whether you ask it in a short prompt or in the middle of a long conversation. The exact choice of scheme affects how far a model can extrapolate and how it handles distant drafts, but all the schemes solve the same core problem: telling attention who came first so that meaning stays intact.

Positional encoding makes attention position-aware, but it also contributes to the mechanism’s biggest weakness. When the model maps every position to every other position, the cost grows fast.

Why does attention get slow with long conversations?

Standard attention compares every token with every other token, so the number of pairwise comparisons is proportional to the square of the context length. For a 1,000-token prompt, the model must compute one million compatibility scores. For 2,000 tokens, it’s four million. This quadratic growth is the reason very long chats feel sluggish, why models have a context window limit, and why engineers spend so much effort optimizing attention kernels.

{
  "type": "bar",
  "title": "Attention Comparisons Grow Quadratically",
  "caption": "Number of pairwise token comparisons per attention layer for different context lengths. Illustrative values based on n squared.",
  "data": [
    {
      "label": "1,024 tokens",
      "value": 1048576
    },
    {
      "label": "2,048 tokens",
      "value": 4194304
    },
    {
      "label": "4,096 tokens",
      "value": 16777216
    },
    {
      "label": "8,192 tokens",
      "value": 67108864
    },
    {
      "label": "16,384 tokens",
      "value": 268435456
    }
  ]
}

The slowness is not just about arithmetic. The real bottleneck is that attention must materialise large intermediate matrices and move them between memory and compute units on a GPU (graphics processing unit). The FlashAttention family of algorithms rearranges the computation to use the GPU’s SRAM more cleverly, tiling and recomputing some values in the backward pass instead of storing them. This can cut memory use and speed up long-sequence training by an order of magnitude. For applications that need even longer contexts, researchers have explored sparse attention patterns that only let tokens attend to a local window plus a few global tokens, or linearised approximations that avoid the full N×N matrix altogether. These are all attempts to get most of the benefit of full attention without the brutal scaling law.

When you actually run a request through a chatbot, all these pieces work together in a pipeline. Let’s walk through what happens during a real generation step.

What does attention actually look like when a chatbot responds?

When you type “Write a short poem about a sleepy cat” and hit enter, the model first converts your text into tokens: “Write”, “ a”, “ short”, “ poem”, “ about”, “ a”, “ sleepy”, “ cat”. Each token gets its embedding plus a positional encoding. As the model begins to produce the poem line by line, attention inside the decoder layers acts like a conductor.

At the moment the model is about to generate the token “dreams”, its attention heads are busy weighing the surrounding context. Some heads will put high weight on the token “poem” to reinforce the task format. Others will attend strongly to “sleepy” and “cat” to keep the subject consistent. Still others will look at the immediate preceding token to decide the probable next syllable. The weights shift token by token, layer by layer, but overall the system routes information so that the generated stream stays on theme. This is why a good chatbot can handle a follow-up question about “fluffy paws” without you repeating “cat” every time. The attention patterns have effectively copied the relevant concept from earlier in the conversation and baked it into the hidden state of the current token.

Now, those bright lines between words that visualisation tools often draw? They are approximations. They show you which tokens received high attention weights, but those weights are not a direct map to what the model actually used. Research has shown that you can change attention patterns dramatically without changing the model output, meaning attention weights are just one piece, a proxy, for the decisions the network makes. So treat those visualizations as hints, not as explanations. The real answer to “how the model decided” is spread across every layer and every head, and it’s still an open research question to trace it fully.

Before we wrap the concepts into a compact reference, let’s address some of the questions engineers inevitably ask once they start peeking under the hood.

Concept	What it does
Query (Q) projection	Encodes what the current token is looking for
Key (K) projection	Encodes what a candidate token offers
Value (V) projection	Encodes the actual information to be mixed
Scaled dot-product	Computes similarity between Q and K, scaled to control saturation
Softmax weighting	Converts similarity scores into a probability distribution
Multi-head attention	Runs several attention instances in parallel, then merges them
Causal mask	Forbids future tokens from being attended to during generation
Cross-attention	Lets decoder tokens attend to encoder outputs in sequence-to-sequence tasks
Positional encoding	Adds sequence order information so attention can distinguish word order
FlashAttention	Algorithm that speeds up attention by keeping data in fast SRAM and reducing memory traffic

Frequently Asked Questions

Q: Is attention computed for every token at every layer?
Yes. In a standard transformer, every token at every layer runs the full multi-head attention block (unless a sparse pattern skips it). This means the raw computational footprint is substantial, but it also allows the model to refine its “focus” at each layer, mixing different levels of abstraction.

Q: Can I use the attention weights to debug why the model gave a wrong answer?
Not reliably. Studies like Attention is not Explanation and The elephant in the interpretability room show that attention weights correlate poorly with feature importance, and alternative attention patterns can often yield the same prediction. Use them as a clue, not as a final verdict. Mechanistic interpretability, which looks at the internal circuits, is more direct but harder.

Q: What’s the difference between self-attention and cross-attention in a chatbot?
In a decoder-only chatbot (the most common LLM), there is no explicit cross-attention; all attention is self-attention over the concatenated conversation history. In classic encoder-decoder models, self-attention operates within the encoder and within the decoder, and cross-attention connects the decoder to the encoder’s output. For a modern chatbot, the “cross” effect is achieved by having the prompt and conversation all in the same sequence, with causal masking separating the assistant’s turns from earlier input.

Q: Why does my long prompt sometimes cause the model to forget the beginning?
Standard attention becomes expensive for sequences much longer than the model’s training context window. Even with optimizations, the architecture may struggle to retain crisp information from thousands of tokens back. Models with positional schemes like RoPE or ALiBi can extrapolate a bit, but the effective range still degrades beyond a point, and key information can get diluted.

Q: Why do models need four sets of projections (Q, K, V, and the output projection) just for attention?
Separating Q, K, and V lets each token wear different hats for different roles. A token might advertise itself as a good source of factual information (high key score in that dimension) while asking for syntactic guidance (a different query pattern). The output projection then mixes the heads’ results back into a unified space that the next layer can use. This separation increases the model’s flexibility and makes training more stable.

Test yourself

You’re building a small assistant that answers questions about a user’s past conversations. The system stores the history as a plain text block and feeds it to the model each time. A user says: “Remember that restaurant I mentioned last week?” The model responds with the correct name. But when the conversation grows past about 8,000 tokens, it starts missing earlier references. How would you explain, using only the attention mechanism, why this happens, and what would you try first?

Answer: The model’s forgetfulness is a direct consequence of the quadratic cost and limited context window of standard attention. As the history grows, the softmax in each attention block must spread a fixed probability mass over an increasing number of tokens. Information from many interactions gets compressed into a finite-size representation, and distant tokens’ contribution can fall below the noise floor. Even if the model’s positional encoding permits very long sequences, the attention weights for tokens 8,000 steps back may become so small that the model effectively ignores them. The first practical step is to use a retrieval-augmented approach: instead of feeding the raw history, extract only the relevant sections with a lightweight search and inject them into the prompt. This shortens the effective sequence length and brings the critical tokens back into the model’s attention spotlight. If you must keep everything, moving to a model fine-tuned with a larger context window and using FlashAttention-2 to manage memory might help, but retrieval is almost always the simpler start.

If you want this kind of breakdown every week, how real systems actually work under the hood without the fluff, subscribe to Internals Decoded at internalsdecoded.com. Next in the series: we stop looking at forward passes and dive into the training loop, where attention and billions of predictions collide to shape the model you chat with.

Sources

Originally published at Internals Decoded. AI internals, explained conversationally.

Tokens: The Currency of AI

Internals Decoded — Wed, 19 Aug 2026 14:04:09 +0000

When you ask a chatbot for a recipe, it does not see words. It sees tokens. Every phrase you type gets chopped into small, numbered pieces. Those pieces are the thing you pay for, the thing that fills the model’s working memory, and the reason your assistant sometimes forgets what you said three paragraphs ago.

A single English word can break into half a dozen tokens in one model and remain whole in another. That mismatch silently inflates your API (application programming interface) bill, truncates your prompt mid-sentence, and explains why your chatbot “forgets” earlier instructions. This article shows you exactly what a token is, where it comes from, and why treating tokens as a first-class resource changes how you design AI systems.

What exactly is a token?

A token is the smallest unit an LLM (large language model) reads. You hand it text. It maps that text to a list of integers. Each integer points to an entry in a fixed vocabulary of subwords, characters, or control symbols.

Think of tokens as the currency of AI. You do not pay for a conversation in words or paragraphs. You pay in tokens. The model processes tokens. It gets confused when tokens run out. Embedding tables, attention layers, and the final output all operate on token indices, not on the characters you typed. When that fact is invisible, you misjudge cost, truncation, and what a model can actually remember.

The vocabulary is built once, at training time, from a huge corpus. It might contain 50,000 to 250,000 entries, each learned to cover frequent strings compactly and rare strings by composition. A token can be a full word like “the”, a morpheme like “ing”, a punctuation mark, or a special marker that tells the model where the system prompt ends. The model never sees raw text. It only ever sees sequences of these integer IDs. tiktoken

Why does a chatbot split "strawberry" into three pieces?

A chatbot splits “strawberry” into multiple tokens because of something called subword tokenization. Instead of keeping every possible word in the vocabulary, the tokenizer learns reusable chunks. Common words stay whole. Rare words get built from smaller parts. That way the model can read any string you throw at it, even typos or made-up words, without ever running into an “unknown” token.

The dominant algorithm is Byte Pair Encoding, or BPE. BPE starts from raw bytes or characters. It scans a massive amount of text and repeatedly merges the most frequent adjacent pair into a new token. After thousands of merges, the vocabulary contains short, frequent sequences like “st”, “raw”, and “berry”. When you feed in “strawberry”, the greedy BPE decoder applies merges in the order they were learned. It finds that “st” matches the earliest applicable merge, then “raw”, then “berry”. So the word becomes three tokens. BPE paper

Here is what that looks like with OpenAI’s tiktoken library for GPT-4:

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4")
tokens = enc.encode("strawberry")
# tokens is something like [72, 75, 76]
for t in tokens:
    print(enc.decode_single_token_bytes(t))
# b'st'  b'raw'  b'berry'

The same logic applies to your recipe. “Give me a recipe for chocolate chip cookies” might tokenize as “Give”, “ me”, “ a”, “ recipe”, “ for”, “ chocolate”, “ chip”, “ cookies”. Each space in the output shows that the tokenizer includes leading spaces as part of the token. That is a byte-level detail that makes token boundaries unintuitive but guarantees every possible byte sequence is representable. OpenAI tokenizer

How does tokenization convert my recipe request into numbers?

Every interaction with a chatbot follows a deterministic pipeline. Text goes in, integer IDs come out. The IDs then drive everything downstream: embedding lookups, positional encodings, and the model’s own computation.

The pipeline works like this:

flowchart LR
    A[Raw text input] --> B[Normalize & byte-encode]
    B --> C[Pre-tokenize on whitespace/punctuation]
    C --> D[Apply BPE merges greedily]
    D --> E[Map subword strings to token IDs]
    E --> F[Sequence of integers fed to model]

First the tokenizer normalizes the string. It might fold case, strip zero-width characters, or enforce one Unicode normalization. Byte-level BPE, used in GPT-2 and later, maps every Unicode code point to its UTF-8 bytes and treats each of the 256 possible bytes as the base alphabet. That step alone guarantees no unknown characters can break the pipeline. byte-level BPE

Next, a pre-tokenizer splits the text into candidate words using whitespace and punctuation. “Give me a recipe for chocolate chip cookies.” becomes a list of seven pieces. Within each piece, the BPE merges run. “chocolate” might stay whole because it is frequent. “cookies” might fragment into “cook” and “ies”. The merge rules are stored as a large hash table; tokenization is a fast, table-driven lookup that never depends on the full vocabulary ordering at runtime.

Finally, each subword string gets mapped to a unique integer ID by a trie or hash map. That sequence of IDs is what the model consumes. The total number of IDs produced is the token count for your request. You can check it yourself with tiktoken before you ever send a prompt to the cloud. Hugging Face tokenizers

Last week we saw that an LLM predicts the next word. Here you can see what “word” really means: it means the next token in this ID sequence. Everything the model learns about grammar, recipes, and style is anchored to these integer indices, not to the English letters you typed.

Why is pricing per token, not per word?

Pricing is per token because the computer works per token. Every token in your prompt triggers a table lookup in the embedding matrix, a row of the positional encoding, and a cascade of matrix multiplications through attention and feed-forward blocks. When the model generates a response, each new token you receive required a full forward pass. Charging per token aligns billing directly with the compute you consume. OpenAI pricing

A “word” is a fuzzy, language-dependent notion. “Pneumonoultramicroscopicsilicovolcanoconiosis” is one English word but could be six tokens. Token counts do not depend on anyone’s definition of a word; they are machine-verifiable. That makes them a sane billing unit across languages, code, logs, and mixture of all three.

Using the recipe example: suppose your prompt “Give me a recipe for chocolate chip cookies” tokenizes to 11 tokens. With a completion that generates another 200 tokens, your total consumption is 211 tokens. If the model charges $0.01 per 1,000 input tokens and $0.03 per 1,000 output tokens, you can predict the exact cost of that recipe interaction before you write a single line of code.

A subtle corollary: phrases that look short but contain rare subwords can cost more than longer, simpler sentences. That is because rare words break into many subword pieces. Here is a side-by-side comparison:

Phrase	Token count (approximate)
“Hi there.”	3 tokens
“Antidisestablishmentarianism”	5 tokens
“let me write you a short poem”	7 tokens

{
  "type": "bar",
  "title": "Token count for different phrases",
  "caption": "Even short phrases can have varying token counts depending on word rarity.",
  "data": [
    {
      "label": "Hi there.",
      "value": 3
    },
    {
      "label": "Antidisestablishmentarianism",
      "value": 5
    },
    {
      "label": "let me write you a short poem",
      "value": 7
    }
  ]
}

The token count, not the character count, is what empties your wallet.

What is the context window, and why does it matter for my chatbot?

The context window is the maximum number of tokens the model can process in a single forward pass. It counts every token: your system prompt, the message history, the current query, and every token the model generates so far. Once the total exceeds that limit, earlier tokens are simply not visible to attention. The model cannot recall them.

If you use a model with a 128k-token context window, you might think you can dump an entire book in the prompt. But if that book is full of rare words that break into many subword tokens, the actual token count might be far higher than the word count suggests. Even a few thousand “words” can eat a surprising fraction of the budget. When the window fills, your recipe from the top of the conversation vanishes, and the model starts answering as if it never saw it. Gemini context window

This is not a failure of memory in a psychological sense. It is a hard resource constraint built into the transformer architecture. Self-attention scales at least quadratically with token count, so the context window is both a limit on what the model can see and a key driver of latency and cost. The wider the window, the more expensive every request becomes.

Engineers who think in tokens check the token count of every prompt before it hits the model. They truncate old messages strategically, summarise lengthy histories, or split long documents into overlapping chunks that each fit comfortably inside the window. In all cases, the first step is to run the exact tokenizer and count.

How does token-level caching save money and latency?

When you send the same system prompt over and over, the model does the same work each time. The matrices that represent the prompt’s tokens are identical from request to request. Modern inference engines exploit that by caching the key-value tensors produced by attention layers for every prefix. These cached tensors are tied directly to the token sequence. If the first N tokens do not change, the model can skip recomputation and jump straight to processing the new tokens. KV caching in Hugging Face

This caching is not a vague concept of “reusing past answers.” It is a mechanical reuse of large, numerical arrays that map one-to-one with token positions. The cache is indexed by token offset in the sequence. Insert a different token at position 0, and the entire prefix cache becomes invalid. Keep the system prompt word-for-word identical, and you can reuse every single one of those tensors.

In the chatbot example, imagine you build a cooking assistant. Its system prompt is a 200-token block that sets the assistant’s tone and gives kitchen safety rules. Without caching, every new recipe request pays to reprocess those 200 tokens. With prompt caching enabled, you pay only for the new user message and the generated tokens. That often cuts latency in half and slashes the per-request cost dramatically. OpenAI prompt caching

graph TD
    A["System prompt tokens"] --> B["Compute KV cache"]
    B --> C["Store cache"]
    D["User message tokens"] --> E["Append to cached prompt"]
    C --> E
    E --> F["Model forward pass"]
    F --> G["Response tokens"]

The takeaway is: when you control your prompt’s token structure carefully, you unlock deep performance wins. Always keep the static prefix at the beginning. Never insert dynamic content before the cached portion. Treat the token sequence as a precious resource whose alignment with the cache saves real money.

Quick reference

Property	Value
Typical token-to-character ratio	~4 characters per token (English)
Typical token-to-word ratio	~¾ word per token
Vocabulary size (GPT-4o)	~100,000 tokens
Context window (GPT-4o)	128,000 tokens
Pricing unit	per 1,000 tokens (input and output priced separately)
Standard special tokens	`<
Byte-level BPE base alphabet	256 bytes

{% raw %}

{
  "type": "stat",
  "title": "Key token metrics",
  "caption": "Approximate ratios and sizes for typical LLMs.",
  "stats": [
    {
      "value": "~0.75",
      "label": "Words per token"
    },
    {
      "value": "~4",
      "label": "Chars per token"
    },
    {
      "value": "50k to 250k",
      "label": "Vocabulary size"
    },
    {
      "value": "Up to 128k",
      "label": "Context window"
    }
  ]
}

Test yourself

You are building a summarisation tool that accepts user-pasted articles. A user submits a 2,500-word news story. You plan to process it with GPT-4o (128k context) and generate a 150-word summary. What token-related concerns should you address before writing the API call? How would you verify your assumptions?

Answer: The 2,500-word input likely expands to roughly 3,300 tokens (assuming ~0.75 tokens per word) plus overhead for a system prompt and instruction, say another 100 tokens. The total prompt tokens are well under the 128k limit, so no truncation is expected. However, the model’s output of ~200 tokens will be charged at a higher rate. You should verify the actual token count using tiktoken before deployment, because domain-specific vocabulary (legal terms, names) might splinter into more subwords and inflate the count. Also check that your integration truncates old conversation turns if the same session later reuses the summary; if the user uploads multiple articles, the accumulated token load could silently push earlier input out of the window.

Frequently Asked Questions

Q: Why does my sentence sometimes cost more tokens than I expected?
Rare words, technical jargon, or non-English text often break into multiple subword tokens. Two sentences of equal character length can differ by 30% or more in token count. Always measure with the actual tokenizer, never estimate from character or word counts.

Q: Do all models use the same tokenizer?
No. Each model family typically trains its own tokenizer on its own data. GPT-4 uses a byte-level BPE with roughly 100k tokens. LLaMA-3 uses a SentencePiece tokenizer with a vocabulary around 128k. Even two BPE tokenizers will segment the same text differently if they were trained on different corpora or with different merge thresholds.

Q: Can I count tokens myself before sending a request?
Yes. Libraries like tiktoken (for OpenAI), Hugging Face tokenizers, and SentencePiece expose the exact same encoding as the model. Call .encode() and check the length. Many API providers also return the token usage in the response, but pre-counting avoids sending over-budget requests in the first place.

Q: What happens if I exceed the context window?
The request will be either rejected by the API or silently truncated. Truncation often keeps the latest tokens and drops the oldest, meaning your earliest instructions vanish. Some providers warn you; others do not. Always design prompts so the total token count never approaches the limit.

Q: Is there any advantage to a smaller vocabulary and longer sequences?
A smaller vocabulary makes the embedding matrix cheaper and guarantees full byte-level coverage without a large number of rare tokens. But it forces the model to process more tokens per message, increasing the quadratic self-attention cost. The right size is an empirical trade-off that different model teams settle on during training.

Keep learning how real systems work

If you want a breakdown like this every week, the engineering behind the APIs you call, subscribe to Internals Decoded. We publish one deep article each week, always sourced from primary code and spec. No recaps. No fluff. Just the internals that make the difference in production.

Sources

Originally published at Internals Decoded. AI internals, explained conversationally.

What an LLM Actually Is (and Why Predicting Words Works)

Internals Decoded — Mon, 17 Aug 2026 13:59:20 +0000

An LLM is a gigantic probability calculator. It takes a sequence of words (tokens), runs them through a stack of transformer layers, and outputs a distribution over the next possible word. Pick one, feed it back, and repeat. That cycle, trained on trillions of tokens, is the entire secret behind chatbots that generate recipes, draft emails, and summarize articles.

The surprising part? There is no database of facts, no planner, and no grammar checker. Every impressive behavior, from solving math problems to mimicking a therapist, emerges from the singular pressure to guess the next word correctly.

Start With the Autocomplete on Your Phone

Before we open the hood, think about the word suggestions that appear above your smartphone keyboard. Type “I’m going to bake” and it might suggest “cookies,” “a cake,” or “bread.” That little strip is a miniature language model, trained on your typing history. It looks at the last few words and ranks the next most probable words. An LLM does the same thing, but with three critical differences: it has seen an enormous fraction of the internet, it can attend to thousands of previous words instead of three, and it uses a far deeper architecture to capture complex patterns.

When you ask a chatbot for a chocolate chip cookie recipe, the model does not search a recipe database. It starts with your prompt as a sequence of word fragments, then repeatedly predicts the next fragment. “Ingredients:” leads to “1 cup” leads to “butter” and so on. The whole recipe rolls out one token at a time, driven only by a massive pattern matcher that has learned from every recipe, blog, and cooking forum in its training data. That is the autocomplete mental model. Now we will see why, at scale, that simple trick produces such rich behavior.

What Does an LLM Actually Compute?

An LLM is a function $f_{θ}$ with billions of parameters $θ$ . It accepts a sequence of discrete symbols (tokens) and returns, for each position, a probability distribution over all possible next tokens Vaswani et al. 2017. The function has two main parts: an embedding layer that turns token IDs into vectors, and a stack of transformer blocks that refine those vectors. At the final layer, a linear projection plus a softmax gives the probabilities.

Tokens: The Model’s Native Vocabulary

Before any neural math happens, raw text is chopped into tokens by a tokenizer such as Byte Pair Encoding (BPE) Sennrich et al. 2016. A token is usually a short subword. For example, “chocolate” might become one token, while “summarizing” might split into “summar” and “izing.” The tokenizer builds a fixed vocabulary (often 50,000 to 100,000 tokens) and maps every text snippet to a sequence of integer IDs. The model never sees characters or words directly; it only processes those IDs.

When you type “write a birthday message for my sister,” the tokenizer converts it to something like [1024, 347, 8912, 15008, 305, 623, 411, 8902]. Those integers are the only input the model receives. The model learns statistical structure at the token granularity, which is why odd spacing or a rare spelling can throw it off SentencePiece. Once you understand that, you see why latency is measured in tokens per second and why the same prompt can behave differently if you add a stray space.

From Integers to Vectors: The Embedding Table

Each token ID is mapped to a dense vector via an embedding table $E \in R^{V \times d_{model}}$ Vaswani et al. 2017. If the vocabulary size $V$ is 50,000 and the hidden dimension $d_{model}$ is 4,096, the table holds 50,000 rows of 4,096 numbers. The embedding for token 1024 is simply its row, a point in a high-dimensional space. At first these rows are random, but during training they shift so that tokens with similar roles end up close together. The distance between the vectors for “cat” and “dog” becomes smaller than the distance between “cat” and “car.”

These raw embeddings get better after they pass through the transformer layers. Each layer enriches a token’s representation by mixing in information from every other token in the sequence. That is how “bank” in “river bank” produces a different final vector than “bank” in “I went to the bank.” The embedding table is the starting point; the stack does the contextualization.

The Autoregressive Distribution

The model’s job is to estimate $p_{θ} (x_{t} ∣ x_{1}, \dots, x_{t - 1})$ for every position $t$ . Over a full sequence, the joint probability factorizes as

p_{θ} (x_{1}, \dots, x_{T}) = t = 1 \prod T p_{θ} (x_{t} ∣ x_{< t}) .

This is the autoregressive assumption: each token depends only on its prefix. At the final layer, a linear “language model head” projects each position’s hidden vector to a vector of logits, one per vocabulary token. A softmax turns those logits into probabilities that sum to one Vaswani et al. 2017.

During training the model sees the true next token and is penalized with cross-entropy loss: $- lo g p_{θ} (true token)$ . The optimizer reduces this loss across billions of examples. At inference, the model picks a token (by greedy argmax, sampling, or top-p) and feeds the extended sequence back into itself. The cycle repeats, generating new text one token at a time.

The whole pipeline looks like this:

flowchart LR
    A[Prompt text] --> B[Tokenizer: integer IDs]
    B --> C[Add token + position embeddings]
    C --> D[Transformer blocks: L layers]
    D --> E[LM head: logits]
    E --> F[Softmax: next-token probabilities]
    F --> G[Sample next token]
    G --> H[Append to input]
    H --> C

For efficiency, most inference engines cache the keys and values from previous positions so each new token only requires one forward pass through the stack rather than recomputing the entire prefix.

Why Does Predicting the Next Word Work So Well?

The secret lies in compression. Minimizing next-token cross-entropy is equivalent to compressing the training corpus as tightly as possible Kaplan et al. 2020. To assign high probability to the correct continuation, the model must internally model syntax, facts, causal relationships, and even the intent behind a prompt. That pressure forces the discovery of reusable patterns, from subject-verb agreement to the steps in a recipe.

Scaling laws show that as we increase parameters, data, and compute, the loss drops predictably Kaplan et al. 2020. The Chinchilla paper later clarified that for a given compute budget, the optimal balance is roughly 20 training tokens per parameter Hoffmann et al. 2022. So bigger models trained on more data simply compress better, and that improved compression delivers abilities that smaller models do not have: translation, coding, basic arithmetic, and in-context learning.

In-context learning is a spectacular example. If you give an LLM a few examples of a new task inside the prompt, and then a new input, it often produces the correct output, even though its weights never changed Brown et al. 2020. Mechanistic interpretability work reveals that circuits called “induction heads” allow the model to detect repeated patterns and copy them forward Olsson et al. 2022. These heads emerge purely from next-token training. So the simple objective, at scale, sculpts general-purpose machinery.

Thus, predicting the next word is not a trivial trick. It is a universal training signal that forces the model to approximate the data-generating distribution, building internal shortcuts for grammar, logic, and pattern recognition.

How Does a Transformer Do This?

The transformer architecture, introduced in “Attention Is All You Need,” processes all tokens in parallel using self-attention Vaswani et al. 2017. A decoder-only LLM stacks many identical blocks. Each block has two main sublayers: a multi-head self-attention layer and a position-wise feed-forward network. Residual connections and layer normalization wrap both.

Self-Attention: Mixing Information Across Positions

For each token, self-attention computes three vectors: query ( $Q$ ), key ( $K$ ), and value ( $V$ ) by multiplying the token’s hidden representation by learned weight matrices. For every pair of positions, the dot product between a query and a key measures compatibility. After scaling by $1/ d_{k}$ and applying a causal mask (so a token cannot see future tokens), a softmax converts those scores into attention weights. The output for a token is a weighted sum of all other tokens’ value vectors, where the weights capture how much each position should influence the current one Vaswani et al. 2017.

Multi-head attention repeats this process with $h$ independent $Q, K, V$ sets in parallel, then concatenates the results. Different heads can attend to different phenomena: one head might track previous occurrences of a word, another might link a pronoun to its antecedent, a third might focus on delimiters in code.

Feed-Forward Network: Per-Token Nonlinear Transformation

After attention mixes information across positions, each token’s vector passes through a two-layer MLP with an activation like GELU: $MLP (x) = W_{2} GELU (W_{1} x + b_{1}) + b_{2}$ . This transformation is applied independently to every position. While attention handles cross-token communication, the MLP introduces nonlinear capacity and can store factual associations in its weights. Studies often find that specific neurons in these MLP layers activate for interpretable concepts Elhage et al. 2021.

Residual Connections and Layer Normalization: The Training Scaffold

Without residual connections, gradients would shrink or explode across dozens of layers. A residual connection simply adds the input of a sublayer to its output: $x + F (x)$ . This gives gradients a direct path during backpropagation He et al. 2016. Layer normalization normalizes each token’s feature vector to zero mean and unit variance, then applies learned scale and shift. It stabilizes activations and accelerates training Ba et al. 2016. Modern LLMs usually place layer norm before each sublayer (pre-norm), which further improves gradient flow Xiong et al. 2020.

Together, residuals and layer norm make it possible to train networks with 96, 120, or more layers, scaling the model’s capacity to capture long-range dependencies.

graph TD
    A["Input"] --> B["Multi-Head Attention"]
    B --> C["Add & Norm"]
    A -.-> C
    C --> D["Feed-Forward MLP"]
    D --> E["Add & Norm"]
    C -.-> E
    E --> F["Output"]

Positional Encodings: Telling the Model About Order

Self-attention is permutation invariant. To inject order, the model adds positional information to the input embeddings. Many modern LLMs use Rotary Position Embedding (RoPE), which encodes positions as rotations of the query and key vectors in 2D subspaces Su et al. 2021. The dot product between a rotated query and a rotated key naturally captures relative distance, decaying with separation. RoPE also lets the model extrapolate to sequence lengths longer than those seen during training.

Without this step, the model would treat “dog bites man” and “man bites dog” identically. Positional encodings make the model sensitive to word order, syntax, and temporal flow.

Quick Reference

Concept	Typical Value or Description
Tokenizer	BPE or SentencePiece; vocabulary size 50k-100k subword tokens
Embedding dimension ( $d_{model}$ )	4096 for a 7B model, up to 8192+ for larger models
Number of layers ( $L$ )	32 for a 7B model, often 80-96 for 70B+ models
Attention heads ( $h$ )	32 per layer (128-dimensional per head when $d_{model} = 4096$ )
Feed-forward intermediate size ( $d_{ff}$ )	Typically 4× $d_{model}$ (e.g., 11008 or 14336)
Position encoding	Rotary Position Embedding (RoPE) in most current LLMs
Activation function	GELU in most transformers (SwigLU in newer variants)
Training objective	Next-token cross-entropy loss
Typical training tokens	Order of 1-2 trillion tokens for a 7B Chinchilla-optimal model
Inference strategy	Autoregressive sampling with KV-caching, top- $p$ and temperature

{
  "type": "stat",
  "title": "Typical LLM Scale (2024 numbers)",
  "caption": "Key dimensions from the quick reference table. Exact numbers vary by model.",
  "stats": [
    {
      "value": "50,000 to 100,000",
      "label": "Vocabulary size"
    },
    {
      "value": "4,096 to 8,192",
      "label": "Embedding dimension"
    },
    {
      "value": "32 to 96",
      "label": "Transformer layers"
    },
    {
      "value": "1 trillion+",
      "label": "Training tokens"
    }
  ]
}

Frequently Asked Questions

Q: Does the model actually understand language, or is it just statistical mimicry?
It builds a compressed, predictive model of token sequences. That internal model captures syntax, semantics, and factual relationships well enough to produce coherent output. Whether that constitutes “understanding” depends on your definition, but the model certainly abstracts patterns far beyond surface-level statistics.

Q: How can it do arithmetic if it only predicts the next word?
Long training on text that includes calculations forces the model to approximate algorithmic reasoning. It learns to attend to digits and mimic the step-by-step process, though it can fail on large numbers because it does not execute true symbolic operations.

Q: Why do bigger models suddenly acquire new abilities?
Scaling laws show a continuous drop in loss, but certain capabilities appear abruptly when the model reaches sufficient capacity to compress the required pattern. This emergent behavior arises because the optimization landscape changes qualitatively at certain scales Wei et al. 2022.

Q: What are tokens, and why not use whole words?
Tokens are the atomic pieces the model sees. Subword tokenization handles rare words and morphology gracefully and keeps the vocabulary size manageable, avoiding the explosion that would come from a full-word vocabulary.

Q: Can an LLM memorize its training data?
Yes. Large models can memorize verbatim passages, especially when data is duplicated. Researchers study this as a privacy and copyright concern, and it is one reason deduplication of training data is important Carlini et al. 2022.

Test yourself

You are building an internal tool that uses an LLM to generate SQL queries from natural language questions. Users report that the model occasionally produces syntactically correct queries that nevertheless reference nonexistent table names. Why might this happen, given that the model never executed a line of SQL during training?

Answer: The model never accesses a live database schema. During training it saw millions of text examples that included SQL snippets and natural language descriptions, often paired with plausible table and column names. It learned to map question patterns to likely SQL structures. When a user asks about “customer revenue,” the model generates a query that looks statistically correct based on training, but it invents table names like customer_revenue or sales_data because it has no ground truth about the actual database. It is fulfilling the prompt under the next-token objective, not executing a grounded lookup. To fix this, you must supply the schema explicitly in the prompt, letting the model’s in-context learning bind to concrete identifiers.

If you want this kind of breakdown every week, the real internals behind the tools you use, subscribe to Internals Decoded at internalsdecoded.com. Next time we will open the black box of training and see how billions of weights actually learn from raw text.

Sources

Originally published at Internals Decoded. AI internals, explained conversationally.