Advanced LLM Engineering
A deep-dive walkthrough of what is under the hood of large language models and how that knowledge changes the way you build with them — covering transformer architecture, self-attention and multi-head attention, positional encoding, tokenization algorithms, context window management, the KV cache, temperature/top-K/top-P sampling, quantization, distillation, fine-tuning vs. RAG, LoRA and QLoRA, small language models, open-weight vs. proprietary models, multimodal LLMs, and model routing and fallback — with C# code for the mechanics, back-of-envelope calculators, and interview-style questions.
Table of Contents
- Introduction
- Transformer Architecture
- Self-Attention and Multi-Head Attention
- Positional Encoding
- Tokenization Algorithms
- Context Window Management
- KV Cache
- Temperature, Top-K, and Top-P Sampling
- Model Quantization
- Model Distillation
- Fine-Tuning vs. RAG
- LoRA and QLoRA
- Small Language Models (SLMs)
- Open-Weight vs. Proprietary Models
- Multimodal LLMs
- Model Routing and Fallback Strategies
- Back-of-the-Envelope Calculators
- Interview-Style Questions
- Common Pitfalls
- Quick Reference Table
- Conclusion
Introduction
Most application developers treat an LLM as a function: text in, text out. That abstraction is enough to build a demo, and not enough to build a product — because nearly every production problem is an internals problem in disguise. Why does the second request with the same prefix come back faster and cheaper? KV cache and prefix caching. Why does the model miscount letters or fumble arithmetic? Tokenization. Why does quality sag when you stuff the context window? Attention's quadratic cost and positional encoding limits. Why does temperature 0 still vary run to run? Sampling and floating-point nondeterminism. Why will a 7B model run on a laptop but a 70B model needs a server? Parameters × bytes per parameter, plus the KV cache. Why is a smaller, fine-tuned model sometimes better than a frontier model? Distillation, LoRA, and task-specific data.
The LLM engineering stack, from the inside out:
TEXT ──> TOKENIZER ──> EMBEDDINGS + POSITIONS ──> [ TRANSFORMER BLOCK x N ] ──> LOGITS ──> SAMPLER ──> next token ──┐
│ self-attention (with KV cache) │
│ feed-forward │
└───────────────────────────── repeat, token by token <───────┘
And around it: quantization (shrink it) · distillation (compress its skill) · LoRA (adapt it cheaply) · RAG (give it knowledge) ·
routing (pick the right model) · fallback (survive failures) · multimodal inputs (see more than text).
This guide connects the mechanics to engineering decisions: what to measure, what to tune, what to route where, and what to avoid. It builds on earlier guides in this series — LLM Application Development (prompting, context, hallucination), AI APIs with .NET (calling models), Embeddings & Vector Search, RAG, AI Production / MLOps (serving, GPUs, caching), and Microsoft.Extensions.AI (IChatClient) — and goes a level deeper on the models themselves. The code is C# where it makes the mechanism concrete (toy attention, a sampler, quantization, a router) and is educational: production inference runs in optimized GPU kernels, not LINQ.
A note on specifics. Model families, parameter counts, context lengths, prices, library APIs (ONNX Runtime GenAI and others), and license terms change constantly. The mechanisms and trade-offs here are durable; verify any named model, API surface, license, or number against current documentation before relying on it. Numeric examples are illustrative.
1. Transformer Architecture
The architecture behind essentially every modern LLM
The transformer (introduced in the 2017 paper "Attention Is All You Need") replaced recurrent networks with a design built from two repeating ideas: attention, which lets every token look at every other token to gather context, and a position-wise feed-forward network, which transforms each token's representation. Stack many identical blocks, and you get a model that processes whole sequences in parallel during training and scales remarkably well with data and compute.
The three families
ENCODER-ONLY (BERT-style) Each token attends to ALL tokens (bidirectional). Produces rich representations; great for classification, embeddings, NER, reranking. Doesn't generate text.
DECODER-ONLY (GPT-style) Each token attends only to PREVIOUS tokens (causal mask). Trained to predict the next token; generates text autoregressively. The architecture of today's chat LLMs.
ENCODER-DECODER (T5-style) An encoder reads the input; a decoder generates output while attending to the encoder. Common in translation and some summarization systems.
Anatomy of a decoder-only LLM
input text
│ tokenizer
v
token IDs [1042, 318, 257, ...]
│ embedding table (vocab_size x d_model)
v
token vectors + positional information (Section 3)
│
v
┌───────────────────── TRANSFORMER BLOCK (repeated N times, e.g. 32 – 120) ─────────────────────┐
│ x ──> Normalize ──> MULTI-HEAD SELF-ATTENTION (causal) ──> + residual ──┐ │
│ ┌─────────────────────────────────────────────────────────────────────┘ │
│ └──> Normalize ──> FEED-FORWARD NETWORK (MLP) ──────────────────────> + residual ──> out │
└─────────────────────────────────────────────────────────────────────────────────────────────────┘
│
v
final normalization ──> LM head (linear layer, d_model → vocab_size) ──> LOGITS (one score per vocabulary token)
│ softmax (with temperature, top-k, top-p — Section 7)
v
next-token probabilities ──> sample ──> append to the sequence ──> repeat
The pieces
EMBEDDINGS A learned table mapping each token ID to a vector of size d_model (for example 4,096). The LM head often reuses (ties) the same matrix.
RESIDUAL CONNECTIONS Each sub-layer's output is ADDED to its input (x + f(x)), letting gradients flow and letting layers refine rather than replace the representation.
NORMALIZATION LayerNorm or RMSNorm stabilizes activations. Modern models usually normalize BEFORE each sub-layer ("pre-norm"), which trains more stably at depth.
ATTENTION Mixes information ACROSS tokens (Section 2).
FEED-FORWARD (MLP) Transforms each token's vector INDEPENDENTLY: expand to a larger hidden size, apply a nonlinearity (modern models often use gated variants such as SwiGLU), project back.
The MLP holds a large share of the parameters (roughly two-thirds in classic designs) and much of the model's "knowledge."
LM HEAD Projects the final vector to one score (logit) per vocabulary entry.
Where the parameters are
Per block (classic design, MLP expansion of 4x):
attention projections (Q, K, V, output): 4 x d²
feed-forward (two matrices, d -> 4d -> d): 8 x d²
≈ 12 x d² parameters per block
Total ≈ 12 x layers x d² (+ embeddings: vocab x d)
Example (GPT-3-scale): 96 layers, d = 12,288 -> 12 x 96 x 12,288² ≈ 174 billion (published figure: ~175B).
Mixture of Experts (MoE)
In an MoE model the single feed-forward network in a block is replaced by many "expert" networks plus a small ROUTER that sends each token to just a few of them (for example 2 of 64).
+ Total parameters (knowledge capacity) can be huge while ACTIVE parameters per token — and therefore compute per token — stay modest.
- ALL experts must still be held in memory; routing adds engineering complexity; serving is harder (expert parallelism, load balancing).
When you see "47B total / 13B active," that's MoE: memory cost follows total parameters, compute cost follows active parameters.
How LLMs are trained (the lifecycle)
1. PRETRAINING Predict the next token over trillions of tokens of text/code. Produces a "base" model with broad knowledge but no instruction-following manners. Enormously expensive.
2. SUPERVISED Fine-tune on curated (instruction, ideal response) examples so the model follows instructions and uses chat formats.
FINE-TUNING (SFT)
3. PREFERENCE Align behavior with human or AI preferences (RLHF — reinforcement learning from human feedback; DPO and related methods optimize preferences more directly): helpfulness, honesty,
TUNING safety, tone.
4. REASONING / RL Newer "reasoning" models add reinforcement learning on verifiable tasks to produce extended step-by-step reasoning before answering — trading latency and cost for accuracy.
Application developers almost never pretrain; they use base-to-chat models as-is, adapt them cheaply (Sections 10-11), or call hosted models.
What this means in practice
- The model is a NEXT-TOKEN PREDICTOR; everything else (reasoning, tool use, JSON) is behavior learned or elicited on top. That's why validation, grounding, and constraints matter.
- Cost scales with PARAMETERS (memory) and with TOKENS processed (compute), which is why prompt length, output length, and model size drive your bill and latency.
- The context window is bounded by how the architecture handles position and by the memory cost of attention (Sections 3, 6).
2. Self-Attention and Multi-Head Attention
The core idea: every token asks "who is relevant to me?"
For each token, self-attention computes a weighted mix of information from the other tokens, where the weights express relevance. It does this with three learned projections of each token's vector:
QUERY (Q) "What am I looking for?"
KEY (K) "What do I contain / what can I be matched on?"
VALUE (V) "What information do I hand over if I'm attended to?"
For a sequence X: Q = X·W_Q K = X·W_K V = X·W_V
Q · Kᵀ
Attention(Q, K, V) = softmax( ────── + mask ) · V
√d_k
STEP BY STEP
1. SCORES Each token's query is dot-producted with every key: how well does token i match token j? (an n x n matrix)
2. SCALE Divide by √d_k so scores don't grow with dimension and saturate the softmax.
3. MASK In decoder-only models, token i may not attend to tokens AFTER it: set those scores to -infinity (the "causal mask").
4. SOFTMAX Turn each row of scores into weights that sum to 1.
5. MIX Output for token i = weighted sum of the VALUE vectors, using those weights.
Attention in C# (a toy, to see the mechanism)
// Educational single-head causal self-attention. Q, K, V are [sequenceLength][headDim].
// Real implementations use batched, fused GPU kernels (and avoid materializing the n x n score matrix).
static float[][] CausalSelfAttention(float[][] Q, float[][] K, float[][] V)
{
int n = Q.Length;
int d = Q[0].Length;
var output = new float[n][];
for (int i = 0; i < n; i++)
{
// Token i may attend to tokens 0..i only (the causal mask)
var scores = new float[i + 1];
for (int j = 0; j <= i; j++)
{
float dot = 0;
for (int k = 0; k < d; k++) dot += Q[i][k] * K[j][k];
scores[j] = dot / MathF.Sqrt(d); // scaled dot product
}
// Softmax (subtract the max for numerical stability)
float max = scores.Max();
float sum = 0;
for (int j = 0; j < scores.Length; j++) { scores[j] = MathF.Exp(scores[j] - max); sum += scores[j]; }
for (int j = 0; j < scores.Length; j++) scores[j] /= sum;
// Weighted sum of value vectors
var o = new float[V[0].Length];
for (int j = 0; j <= i; j++)
for (int k = 0; k < o.Length; k++) o[k] += scores[j] * V[j][k];
output[i] = o;
}
return output;
}
Multi-head attention: several "views" in parallel
One attention pattern can't capture everything — syntax, coreference ("it" → "the cat"), proximity, formatting, and long-range dependencies all matter. Multi-head attention runs several attention operations ("heads") in parallel, each with its own Q/K/V projections over a smaller subspace, then concatenates the results and mixes them with an output projection.
d_model = 4,096, heads h = 32 -> each head works in head_dim = d_model / h = 128 dimensions.
head_1 .. head_32 each compute attention independently -> concatenate (back to 4,096) -> output projection W_O.
Different heads learn different roles (interpretability research finds heads that track previous tokens, match brackets, copy earlier text, and more).
The cost of attention
COMPUTE AND MEMORY: the score matrix is n x n -> cost grows QUADRATICALLY with sequence length n.
Double the context and attention work roughly quadruples (the MLP part grows linearly).
This is the fundamental reason long contexts are expensive and slow, and why so much engineering goes into attention efficiency.
Attention variants you will meet
MULTI-QUERY ATTENTION (MQA) All query heads SHARE one key/value head -> the KV cache (Section 6) shrinks ~h-fold; slight quality cost.
GROUPED-QUERY ATTENTION (GQA) Query heads are split into groups, each group sharing one K/V head. A middle path (for example 32 query heads, 8 KV heads) that most modern open models use:
near-MHA quality with a much smaller KV cache and faster decoding.
MULTI-HEAD LATENT ATTENTION Compresses keys/values into a low-dimensional latent to shrink the cache even further (used in some recent models).
FLASHATTENTION NOT an approximation — an exact attention algorithm reorganized to minimize slow GPU memory traffic (tiling in fast on-chip memory), giving large speedups and
avoiding the n x n matrix in memory. Standard in modern training and serving.
SLIDING-WINDOW / LOCAL / SPARSE Each token attends only to a recent window (or a pattern), cutting cost from quadratic toward linear, at the price of direct long-range access (often combined
with some global layers or "attention sinks").
CROSS-ATTENTION Queries from one sequence attend to keys/values from ANOTHER (decoder over encoder output; text over image features in some multimodal designs).
3. Positional Encoding
Attention alone has no sense of order
Self-attention treats its input as a set: shuffle the tokens and, without extra information, every token's output is just as valid. But "dog bites man" ≠ "man bites dog." The model must be told where each token sits. That is the job of positional encoding.
Approaches
SINUSOIDAL (ABSOLUTE) The original transformer added fixed sine/cosine waves of different frequencies to each token embedding:
PE(pos, 2i) = sin(pos / 10000^(2i/d))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d))
Each position gets a unique pattern; no learned parameters.
LEARNED ABSOLUTE A trainable embedding per position (used by earlier GPT models). Simple, but can't extrapolate beyond the lengths seen in training.
RELATIVE POSITION BIAS Encode the DISTANCE between tokens (added to attention scores), as in T5-style models.
ALiBi Add a fixed penalty to attention scores that grows linearly with distance, with no position embeddings at all — good length extrapolation.
ROTARY (RoPE) Rotate the query and key vectors by an angle proportional to their position. Because the dot product of two rotated vectors depends on the DIFFERENCE of their rotation
angles, attention scores naturally depend on RELATIVE position. The dominant choice in modern open models.
Sinusoidal encoding in C
static float[][] SinusoidalPositionalEncoding(int maxPositions, int dModel)
{
var pe = new float[maxPositions][];
for (int pos = 0; pos < maxPositions; pos++)
{
pe[pos] = new float[dModel];
for (int i = 0; i < dModel; i += 2)
{
double angle = pos / Math.Pow(10000, (double)i / dModel);
pe[pos][i] = (float)Math.Sin(angle);
if (i + 1 < dModel) pe[pos][i + 1] = (float)Math.Cos(angle);
}
}
return pe;
}
Rotary embeddings (RoPE) in C
// Rotate consecutive pairs of dimensions by a position-dependent angle. Applied to Q and K (not V) before the dot product.
static void ApplyRope(float[] vector, int position, float theta = 10000f)
{
int d = vector.Length;
for (int i = 0; i < d; i += 2)
{
float frequency = MathF.Pow(theta, -(float)i / d); // low dimensions rotate fast, high dimensions slowly
float angle = position * frequency;
float cos = MathF.Cos(angle), sin = MathF.Sin(angle);
float x0 = vector[i], x1 = vector[i + 1];
vector[i] = x0 * cos - x1 * sin;
vector[i + 1] = x0 * sin + x1 * cos;
}
}
Why this matters for context length
A model only reliably handles positions it saw during training. Pushing beyond that range degrades quality. Techniques extend context after training:
POSITION INTERPOLATION Compress positions so a longer sequence maps into the trained range, then briefly fine-tune.
NTK-AWARE / YaRN SCALING Rescale RoPE frequencies so high-frequency detail is preserved while low frequencies stretch.
LONG-CONTEXT TRAINING Continue training on longer sequences.
Advertised context length is therefore a property of the TRAINING and POSITION SCHEME, not just a config number — and quality at the far end of a long window is often weaker than the headline suggests
(see "lost in the middle," Section 5).
4. Tokenization Algorithms
Models read tokens, not characters
A tokenizer converts text into a sequence of integer IDs drawn from a fixed vocabulary (commonly 32,000 to 256,000 entries), and converts IDs back to text. It sits outside the neural network and is trained separately — and its design shapes cost, multilingual fairness, arithmetic ability, and quirks.
Why not just characters or words?
CHARACTERS Tiny vocabulary, no unknown words — but sequences become very long (expensive attention) and each token carries little meaning.
WORDS Short sequences — but a huge vocabulary, no way to represent unseen words, typos, or new names ("out-of-vocabulary").
SUBWORDS The compromise: common words are single tokens; rare words split into reusable pieces ("un" + "believ" + "ably"). Open vocabulary, moderate length. All modern LLMs use subword tokenization.
The main algorithms
BYTE-PAIR ENCODING (BPE)
Start from individual symbols (characters or BYTES). Repeatedly find the most frequent adjacent pair in the training corpus and MERGE it into a new symbol. Stop at the target vocabulary size.
Encoding applies the learned merges in order. BYTE-LEVEL BPE (GPT-2 onward) starts from the 256 bytes, so ANY text — any language, emoji, binary — can be encoded with no "unknown" token.
Used by GPT-family models (via the tiktoken implementation), Llama 3 and others.
WORDPIECE
Like BPE, but chooses merges that maximize the likelihood of the training data rather than raw frequency. Continuation pieces are marked ("##ing"). Used by BERT-style models.
UNIGRAM LANGUAGE MODEL
Start with a LARGE candidate vocabulary and repeatedly PRUNE the pieces that hurt corpus likelihood the least. Encoding finds the most probable segmentation; it can also SAMPLE alternative
segmentations (useful as regularization). Implemented in SentencePiece.
SENTENCEPIECE
A library/framework (supporting BPE and Unigram) that treats text as a raw character stream — no language-specific pre-splitting; spaces become a visible marker (▁). Used by Llama 2, T5, Gemma and many others.
PRE-TOKENIZATION
Most tokenizers first split text with a regular expression (separating words, numbers, punctuation, whitespace) so merges don't cross certain boundaries. The regex is part of the tokenizer's identity.
A toy BPE learner in C
// Learns merge rules from a word list — the heart of BPE. (Educational: real trainers work on bytes, at corpus scale, with careful data structures.)
public static List<(string A, string B)> LearnBpeMerges(IEnumerable<string> words, int numMerges)
{
// Represent each word as space-separated symbols plus an end-of-word marker, with its frequency
var vocab = words.GroupBy(w => w).ToDictionary(
g => string.Join(" ", g.Key.Select(c => c.ToString())) + " </w>",
g => g.Count());
var merges = new List<(string A, string B)>();
for (int step = 0; step < numMerges; step++)
{
// Count every adjacent symbol pair, weighted by word frequency
var pairCounts = new Dictionary<(string, string), int>();
foreach (var (word, freq) in vocab)
{
string[] symbols = word.Split(' ');
for (int i = 0; i < symbols.Length - 1; i++)
{
var pair = (symbols[i], symbols[i + 1]);
pairCounts[pair] = pairCounts.GetValueOrDefault(pair) + freq;
}
}
if (pairCounts.Count == 0) break;
// Merge the most frequent pair everywhere it occurs
var best = pairCounts.MaxBy(kv => kv.Value).Key;
merges.Add(best);
string pattern = $@"(?<!\S){Regex.Escape(best.Item1 + " " + best.Item2)}(?!\S)"; // whole-symbol matches only
string replacement = best.Item1 + best.Item2;
vocab = vocab.ToDictionary(kv => Regex.Replace(kv.Key, pattern, replacement), kv => kv.Value);
}
return merges;
}
// LearnBpeMerges(["low", "low", "lower", "newest", "newest", "widest"], 10) learns merges like ("e","s") -> "es" -> ("es","t") -> "est" -> ...
// i.e. frequent character sequences become single vocabulary entries.
Using real tokenizers in .NET
// dotnet add package Microsoft.ML.Tokenizers
using Microsoft.ML.Tokenizers;
Tokenizer tokenizer = TiktokenTokenizer.CreateForModel("gpt-4o"); // BPE tokenizer matching an OpenAI model family
IReadOnlyList<int> ids = tokenizer.EncodeToIds("Tokenization matters for cost.");
int count = tokenizer.CountTokens("Tokenization matters for cost.");
string roundTrip = tokenizer.Decode(ids);
foreach (EncodedToken token in tokenizer.EncodeToTokens("unbelievably", out string? _))
Console.WriteLine($"{token.Id,8} '{token.Value}'"); // see exactly how a word is split into pieces
(The Microsoft.ML.Tokenizers package also supports SentencePiece-style (Llama) and WordPiece (BERT) tokenizers; check its current API for the version you install. Always use the tokenizer that matches the model — a mismatched tokenizer silently produces garbage.)
Consequences you can see in production
COST AND LIMITS ARE IN TOKENS "1,000 words" is not a fixed price: ~750 tokens in English, noticeably more in many other languages and in code.
THE MULTILINGUAL "TOKEN TAX" Tokenizers trained mostly on English split other languages (and scripts such as Devanagari, Thai, Arabic) into more, smaller pieces — so the same content costs more, fills the
context faster, and can run slower. Measure token counts on YOUR languages before pricing or sizing.
ARITHMETIC AND SPELLING The model sees tokens, not letters or digits as such: "strawberry" may be 2-3 tokens, so counting letters is genuinely hard; numbers split inconsistently ("12345" vs "1 2345"),
hurting arithmetic. Do these tasks in code (tools).
WHITESPACE AND CASE MATTER " hello" and "hello" and "Hello" are different tokens; leading spaces and trailing whitespace in prompts change behavior and token counts.
CODE AND STRUCTURED TEXT Indentation, long identifiers, and JSON punctuation consume many tokens; minified or compact formats can save cost.
SPECIAL TOKENS AND CHAT TEMPLATES Roles (system/user/assistant), end-of-turn, and tool markers are special tokens inserted by a chat template. Getting the template wrong for an open-weight model
(a mismatched template) is a classic cause of poor output. Never let user text inject special-token strings.
GLITCH TOKENS Rare or undertrained tokens can cause erratic behavior.
TOKEN HEALING / BOUNDARIES Cutting a prompt mid-word (for completion) can place a token boundary the model rarely saw in training and degrade results.
Tokenization checklist for engineers
- Count tokens with the right tokenizer BEFORE sending (budgets, rate limits, cost estimates).
- Test token efficiency for your languages and data formats; consider compact formats for large structured payloads.
- Keep tokenizer, vocabulary, and chat template pinned with the model version (see AI Production / MLOps, Section 2).
- Don't ask the model to do character-level or digit-level work it can't see — give it a tool.
5. Context Window Management
The context window is bounded — and "effective" context is smaller than advertised
The context window is the maximum number of tokens (prompt + generated output) a model can handle in one request. The basics — budgeting, trimming, summarizing, retrieval — are covered in LLM Application Development, Section 6, and AI APIs with .NET, Section 8. This section covers the advanced realities behind them.
What actually limits context
POSITION SCHEME The model is reliable only over position ranges seen in (long-context) training (Section 3).
COMPUTE Attention cost grows quadratically with length; prefill for a 100k-token prompt is slow and expensive.
MEMORY The KV cache grows linearly with length x concurrent sequences (Section 6) — often the binding constraint when serving many users.
ATTENTION QUALITY More tokens means attention is spread thinner, and irrelevant text competes with relevant text.
Effective context versus the headline number
NEEDLE-IN-A-HAYSTACK TESTS Check whether a model can retrieve one planted fact from a long document. Many models do well — but a single easy lookup doesn't predict performance on tasks that need to
combine many pieces, reason over the whole text, or ignore distractors.
"LOST IN THE MIDDLE" Research has shown that information in the middle of a long prompt is often used less reliably than information near the beginning or end. Improved in newer models, but
don't assume it's gone.
CONTEXT ROT / DISTRACTION Quality on a fixed task can DEGRADE as irrelevant or conflicting text is added — even well within the window.
INSTRUCTION DILUTION Instructions at the top can lose force after tens of thousands of tokens of content; restate critical rules near the end.
SO: measure YOUR task at YOUR typical and maximum context lengths. The usable window is whatever length your evaluation shows holds quality.
Long context vs. RAG
LONG CONTEXT ("just put it all in") + Simple; the model sees everything; good for tasks needing the whole document (summarize, review, cross-reference).
- Cost and latency scale with length on EVERY request (mitigated by prompt caching); quality can degrade; no access control per passage; bounded by the window.
RAG (retrieve what's relevant) + Cheap per request, scales to huge corpora, supports citations and per-user permissions and deletion.
- Retrieval can miss; needs infrastructure and evaluation (see the RAG guide).
HYBRID Retrieve broadly, then feed a generous (not unlimited) context; cache the stable prefix.
Choose per task; large windows make "stuff it in" viable for moderately sized corpora, but they don't remove the need for evaluation.
Strategies for managing context
BUDGET EXPLICITLY window = system + tools + history + retrieved context + user input + RESERVED OUTPUT + safety margin. Reserve output tokens first.
RANK AND PACK Fill the budget with the highest-value items first (below).
COMPACT HISTORY Summarize older turns; keep recent turns verbatim; keep a small structured scratchpad (goals, decisions, key facts).
TRIM TOOL OUTPUT Tool results dominate agent contexts — return only what's needed; truncate and summarize large results.
PLACE WELL Stable instructions first (also enables prefix caching), the question and critical constraints near the END, reference material clearly delimited.
CACHE THE PREFIX Keep the static prefix byte-identical so providers (and your own servers) can reuse its KV cache (Section 6).
ISOLATE WITH SUB-AGENTS Give sub-tasks fresh, small contexts and take back summaries (see AI Agents, Section 7).
RE-ASK, DON'T RE-STUFF For long documents, run focused queries over chunks (map-reduce) rather than one giant prompt (see NLP with .NET, Section 5).
MONITOR Track input tokens per request, context utilization, truncation (finish_reason = length), and quality vs. context length (see AI Observability).
A priority-based context packer
public sealed record ContextItem(string Id, string Text, int Priority, int Tokens);
public static class ContextPacker
{
// Choose the most valuable items that fit the token budget. (Greedy by priority; ties favor smaller items.)
public static IReadOnlyList<ContextItem> Pack(IEnumerable<ContextItem> candidates, int budgetTokens)
{
var chosen = new List<ContextItem>();
int used = 0;
foreach (ContextItem item in candidates.OrderByDescending(i => i.Priority).ThenBy(i => i.Tokens))
{
if (used + item.Tokens > budgetTokens) continue; // skip items that don't fit; smaller, lower-priority ones may still fit
chosen.Add(item);
used += item.Tokens;
}
return chosen;
}
}
// Budget = window - reservedOutput - system - tools - safetyMargin
int available = 128_000 - 4_000 - systemTokens - toolTokens - 1_000;
var packed = ContextPacker.Pack(retrievedPassages.Select(p => new ContextItem(p.Id, p.Text, Priority: p.RerankScore >= 0.8 ? 3 : 2, Tokens: tokenizer.CountTokens(p.Text))), available);
6. KV Cache
Why generation would be hopelessly slow without it
An LLM generates one token at a time: each new token needs attention over all previous tokens. Naively, producing token t would recompute keys and values for every earlier token — work that was already done for token t-1. The KV cache stores each layer's key and value vectors for all processed tokens so each decoding step only computes K and V for the new token and reads the rest from memory.
PREFILL (processing the prompt): all prompt tokens are processed IN PARALLEL; K and V are computed and stored. Compute-bound. Determines TIME TO FIRST TOKEN (TTFT).
DECODE (generating output): one token per step; compute Q, K, V for just that token; attend over the cached K/V; append the new K/V. Memory-bandwidth-bound (the whole model's weights and
the growing cache must be read for every single token). Determines TIME PER OUTPUT TOKEN.
Without a cache, decode work per token grows with sequence length; with it, each step does a constant amount of new projection work plus a read of the cache.
How big is the cache?
KV cache bytes = 2 (K and V) x layers x kv_heads x head_dim x sequence_length x batch_size x bytes_per_element
Example — a 7B-class model with full multi-head attention (32 layers, 32 KV heads, head_dim 128, FP16 = 2 bytes):
per token: 2 x 32 x 32 x 128 x 2 B = 524,288 B = 0.5 MiB
4,096 tokens, 1 sequence: 2 GiB -> 16 concurrent sequences: 32 GiB (more than the ~13 GiB of weights!)
Same model with GROUPED-QUERY attention (8 KV heads): per token 128 KiB; 4,096 tokens = 0.5 GiB per sequence.
A 70B-class GQA model (80 layers, 8 KV heads, head_dim 128, FP16): 320 KiB per token; 8,192 tokens ≈ 2.5 GiB per sequence.
For serving, the KV cache — not the weights — is often what limits concurrency: GPU memory = weights + (KV cache per sequence x concurrent sequences) + activations. This is why GQA, cache quantization, and cache management dominate modern serving design.
Techniques that reduce or manage the cache
GROUPED-QUERY / MULTI-QUERY ATTENTION Fewer KV heads -> proportionally smaller cache (Section 2). Chosen at model-design time.
KV-CACHE QUANTIZATION Store keys/values in FP8 or INT8 instead of FP16 -> roughly half the memory, small quality impact (measure it).
PAGEDATTENTION (vLLM and similar) Manage the cache in fixed-size blocks like virtual-memory pages instead of one contiguous buffer per sequence -> far less fragmentation and waste -> more concurrent sequences.
PREFIX CACHING Reuse the cached K/V of a shared prefix (system prompt, tool definitions, a reference document) across requests, so only the NEW suffix needs prefill.
This is what providers' "prompt caching" discounts reflect.
SLIDING WINDOW / ATTENTION SINKS Keep only recent tokens (plus a few initial "sink" tokens) for streaming-style use with bounded memory, accepting loss of distant context.
EVICTION / COMPRESSION Drop or merge cache entries judged unimportant; research-stage trade-offs between memory and fidelity.
OFFLOADING Move cold cache blocks to CPU memory or disk and fetch as needed (adds latency).
Consequences for how you build
PUT STATIC CONTENT FIRST A prompt whose first N tokens are identical across requests can reuse the cache for that prefix — big TTFT and cost wins. Any change in the prefix (a timestamp, a reordered tool
list, a per-request ID at the top) invalidates the reuse from that point on. Variable content goes LAST.
LONG PROMPTS COST TTFT Prefill time grows with prompt length; a huge prompt delays the first token even if the answer is short.
OUTPUT LENGTH COSTS DECODE Every generated token is a full pass through the model: long answers are slow regardless of prompt size.
CONCURRENCY IS MEMORY-BOUND Self-hosted: size GPU memory for weights + cache x expected concurrency; this, not FLOPs, often decides how many users a GPU serves.
HOSTED MODELS You don't manage the cache, but you pay for it: cached-prefix discounts, TTLs, and minimum prefix lengths are the visible effects (see AI Observability, Section 3).
Inference engineering around the cache (briefly)
CONTINUOUS BATCHING Instead of waiting for a whole batch to finish, the server adds new requests and removes finished ones at every decoding step -> much higher GPU utilization (vLLM, TGI, TensorRT-LLM).
CHUNKED PREFILL Split long prompts so they don't stall other requests' decoding.
SPECULATIVE DECODING A small "draft" model proposes several tokens; the big model VERIFIES them in one parallel pass, accepting the matching prefix. Same output distribution, faster decoding when the
draft is usually right (and a natural use for a small language model, Section 12).
TENSOR / PIPELINE PARALLELISM Split a model that doesn't fit on one GPU across several.
DECODE IS BANDWIDTH-BOUND Tokens/sec per sequence ≈ memory bandwidth / bytes of weights read per token (Section 16) — which is why quantization (Section 8) speeds decoding, not just shrinks memory.
7. Temperature, Top-K, and Top-P Sampling
From logits to a chosen token
The final layer outputs logits: one raw score per vocabulary token. Sampling turns them into a probability distribution and picks the next token. The knobs you set in an API (temperature, top_p, sometimes top_k) control that last step.
softmax with temperature T: p_i = exp(z_i / T) / Σ_j exp(z_j / T)
Example logits [2.0, 1.0, 0.1] for three candidate tokens:
T = 0.5 -> [0.864, 0.117, 0.019] sharper: the leader dominates
T = 1.0 -> [0.659, 0.242, 0.099] the model's "natural" distribution
T = 2.0 -> [0.502, 0.304, 0.194] flatter: unlikely tokens get a real chance
T -> 0 greedy: always the highest-probability token (deterministic, in principle)
The controls
GREEDY (T = 0) Always pick the single most likely token. Deterministic in theory; can be repetitive and can get stuck in loops. (In practice, floating-point and batching effects on GPUs can
still cause tiny variations — "temperature 0" is not a guarantee of identical outputs.)
TEMPERATURE Rescales the distribution BEFORE sampling. Low = focused and consistent (extraction, code, factual Q&A); high = varied and surprising (brainstorming, creative writing).
TOP-K Keep only the K most probable tokens, renormalize, sample among them. Simple but blunt: K=40 is too many when the model is certain and too few when many continuations are reasonable.
TOP-P (NUCLEUS) Keep the SMALLEST set of tokens whose cumulative probability reaches P (for example 0.9), renormalize, sample. Adapts to the distribution's shape: few candidates when the model is
confident, many when it's uncertain. With the example distribution at T=1 and P=0.9, only the top two tokens (0.659 + 0.242 ≈ 0.90) survive.
MIN-P Keep tokens whose probability is at least a fraction of the TOP token's probability — another adaptive truncation offered by some inference engines.
PENALTIES Frequency penalty (discourage tokens in proportion to how often they've appeared), presence penalty (discourage any token that has appeared), and repetition penalties reduce loops.
LOGIT BIAS / MASKING Add or subtract from specific tokens' logits to encourage or forbid them. Constrained decoding (JSON schema, grammars) masks every token that would violate the format at each step —
the mechanism behind structured outputs.
BEAM SEARCH Tracks several candidate sequences and picks the highest-scoring overall. Common in translation; rarely used for open-ended chat (it favors bland, repetitive text).
SEED Some APIs accept a seed to make sampling reproducible "best effort."
A sampler in C
// Educational sampler: temperature -> (top-k) -> (top-p) -> sample. Real engines use partial selection on GPU instead of a full sort over a 100k+ vocabulary.
public static int SampleNextToken(ReadOnlySpan<float> logits, float temperature, int topK, float topP, Random rng)
{
int n = logits.Length;
if (temperature <= 0) // greedy
{
int best = 0;
for (int i = 1; i < n; i++) if (logits[i] > logits[best]) best = i;
return best;
}
// 1. Temperature-scaled softmax
float max = float.NegativeInfinity;
foreach (float z in logits) max = Math.Max(max, z);
var probs = new float[n];
float sum = 0;
for (int i = 0; i < n; i++) { probs[i] = MathF.Exp((logits[i] - max) / temperature); sum += probs[i]; }
for (int i = 0; i < n; i++) probs[i] /= sum;
// 2. Rank tokens by probability
int[] order = Enumerable.Range(0, n).OrderByDescending(i => probs[i]).ToArray();
// 3. Top-k: at most k candidates (0 = disabled)
int keep = topK > 0 ? Math.Min(topK, n) : n;
// 4. Top-p: the smallest prefix whose cumulative probability reaches p
float cumulative = 0;
int cutoff = 0;
while (cutoff < keep)
{
cumulative += probs[order[cutoff]];
cutoff++;
if (cumulative >= topP) break;
}
// 5. Renormalize over the surviving candidates and sample
float kept = 0;
for (int i = 0; i < cutoff; i++) kept += probs[order[i]];
float r = (float)rng.NextDouble() * kept, acc = 0;
for (int i = 0; i < cutoff; i++)
{
acc += probs[order[i]];
if (r <= acc) return order[i];
}
return order[cutoff - 1];
}
Practical guidance
CHANGE ONE: adjust temperature OR top_p, not both at once — they overlap in purpose, and changing both makes behavior hard to reason about.
TASKS: extraction, classification, code, tool calls, structured output -> low temperature (0-0.3). Chat/Q&A -> low to moderate. Creative writing, brainstorming -> higher.
STRUCTURED OUTPUT: use schema-constrained decoding (structured outputs) rather than hoping low temperature yields valid JSON.
REASONING MODELS: many restrict or ignore sampling parameters and use other controls (reasoning effort); check the model's documentation.
DETERMINISM: temperature 0 + a seed reduces but doesn't eliminate variation; design evaluation and caching with that in mind (repeat runs; tolerance bands).
DIVERSITY: if you need N different candidates (for self-consistency or ranking), sample at moderate temperature rather than re-running greedy.
EVALUATE: tune on a representative test set, not a handful of prompts.
8. Model Quantization
Trading numeric precision for memory and speed
A model's weights are numbers. Quantization stores them with fewer bits — shrinking memory, reducing the data that must be read per token (decode speed is bandwidth-bound), and enabling models to fit on cheaper hardware — at a (usually small) cost in accuracy.
PRECISION BYTES/PARAM 7B MODEL (weights only)
FP32 4 ≈ 26 GiB
FP16 / BF16 2 ≈ 13 GiB <- the common "full" precision for inference
INT8 / FP8 1 ≈ 6.5 GiB
INT4 (4-bit) 0.5 ≈ 3.3 GiB (+ small overhead for scales)
How it works: scale and round
SYMMETRIC INT8 QUANTIZATION: scale = max(|x|) / 127 q = round(x / scale) (clamped to -127..127) x̂ = q x scale (dequantize)
AFFINE (ASYMMETRIC): q = round(x / scale) + zero_point — handles ranges not centered on zero.
The rounding error per value is at most scale/2; the question is how much that error hurts the model's output.
public static (sbyte[] Quantized, float Scale) QuantizeInt8(float[] values)
{
float maxAbs = values.Select(MathF.Abs).Max();
float scale = maxAbs == 0 ? 1f : maxAbs / 127f;
sbyte[] q = values.Select(v => (sbyte)Math.Clamp((int)MathF.Round(v / scale), -127, 127)).ToArray();
return (q, scale);
}
public static float[] Dequantize(sbyte[] q, float scale) => q.Select(v => v * scale).ToArray();
// Group-wise: each block of (say) 64 weights gets its OWN scale, so one outlier only degrades its own group instead of the whole tensor
public static (sbyte[] Q, float[] Scales) QuantizeInt8Grouped(float[] values, int groupSize = 64)
{
var q = new sbyte[values.Length];
var scales = new float[(values.Length + groupSize - 1) / groupSize];
for (int g = 0; g < scales.Length; g++)
{
int start = g * groupSize, len = Math.Min(groupSize, values.Length - start);
float maxAbs = 0;
for (int i = 0; i < len; i++) maxAbs = Math.Max(maxAbs, MathF.Abs(values[start + i]));
scales[g] = maxAbs == 0 ? 1f : maxAbs / 127f;
for (int i = 0; i < len; i++)
q[start + i] = (sbyte)Math.Clamp((int)MathF.Round(values[start + i] / scales[g]), -127, 127);
}
return (q, scales);
}
The design space
WHAT IS QUANTIZED
WEIGHT-ONLY (e.g. "W4A16") Weights stored in 4/8 bits, dequantized on the fly; activations stay in 16-bit. The most common for LLM inference: big memory win, modest quality loss.
WEIGHTS + ACTIVATIONS ("W8A8") Both in 8-bit so matrix multiplies run in integer/FP8 hardware — faster compute, but activations have OUTLIERS that make this harder (techniques such as
SmoothQuant and LLM.int8() handle them).
KV CACHE Quantize cached keys/values (Section 6) to raise concurrency.
GRANULARITY
PER-TENSOR (coarse), PER-CHANNEL, PER-GROUP (for example groups of 32/64/128 weights) — finer granularity means better accuracy at slightly more storage for scales.
WHEN
POST-TRAINING QUANTIZATION (PTQ) Quantize a finished model, often with a small calibration dataset. Cheap and by far the most common for LLMs.
QUANTIZATION-AWARE TRAINING (QAT) Simulate quantization during training so the model adapts. Better quality at very low bit-widths; far more expensive.
POPULAR METHODS AND FORMATS (names you will encounter)
GPTQ, AWQ Post-training 4-bit weight quantization methods that choose roundings/scales to minimize output error, protecting the most important weights.
bitsandbytes (NF4, int8) Library supporting 8-bit and 4-bit "NormalFloat" quantization — the basis of QLoRA (Section 11).
GGUF (llama.cpp k-quants) File format and mixed-precision quantization schemes for CPU/consumer-GPU inference.
FP8 8-bit floating point supported by recent datacenter GPUs for both weights and activations.
ONNX Runtime quantization Tooling to quantize ONNX models (INT8 for classic models; INT4 weight-only paths for LLMs via ONNX Runtime GenAI model builders).
What you gain and risk
GAINS Memory (fit a bigger model, or more concurrent sequences), speed (less data to read per token; faster integer math on supported hardware), cost, and edge/CPU deployment.
RISKS Quality degradation — small at 8-bit, noticeable but often acceptable at 4-bit for larger models, and increasingly severe below that; larger models tolerate quantization better than small ones.
Some tasks (math, code, long-context reasoning, low-resource languages, precise formatting) degrade sooner than general chat. Outlier weights/activations are the usual culprit.
How to decide — always measure
1. Start from the highest-quality precision you can afford; step down (FP16 -> INT8 -> INT4) while measuring.
2. EVALUATE ON YOUR TASK with your evaluation set (not just perplexity, which can look fine while task accuracy drops). Include tail cases: long inputs, structured output, tool calls, other languages.
3. Compare cost per inference and latency at each level (see AI Production / MLOps, Section 10) — a quantized model may let you drop to a cheaper GPU or CPU.
4. Prefer a larger model quantized to 4-bit over a smaller model at 16-bit when memory is the constraint — this often wins on quality per byte, but verify for your task.
5. Version quantized artifacts like any model (method, bit-width, group size, calibration data) and record them on traces.
9. Model Distillation
Teaching a small model to behave like a big one
Distillation trains a small student model to reproduce the behavior of a large teacher, so you get much of the teacher's skill at a fraction of the size, latency, and cost. It's the main way "small" models punch above their weight.
Flavors
CLASSIC (LOGIT) DISTILLATION The student learns from the teacher's full output distribution ("soft targets") rather than just the correct label. A temperature T softens both distributions
so the student also learns the teacher's relative beliefs about wrong answers ("dark knowledge": a cat photo is more like a dog than a truck).
L = α · CE(y, student) + (1 − α) · T² · KL( teacher_T ‖ student_T )
SEQUENCE-LEVEL / SYNTHETIC-DATA DISTILLATION The most common approach for LLMs today: have the teacher GENERATE training data — answers, rationales, tool-use traces, classifications — and fine-tune the
student on it with ordinary supervised learning. No access to teacher logits is needed, so it works even when the teacher is an API.
REASONING-TRACE DISTILLATION Train the student on the teacher's step-by-step reasoning, transferring problem-solving patterns, not just final answers.
FEATURE / HIDDEN-STATE DISTILLATION Match intermediate representations (used for models like DistilBERT); requires access to both models' internals.
ON-POLICY DISTILLATION The student generates, the teacher scores or corrects, and the student learns from the feedback — closing the gap between training and inference behavior.
The distillation loss in C
// KL divergence between teacher and student next-token distributions, both softened by temperature T
public static double DistillationLoss(float[] teacherLogits, float[] studentLogits, int trueTokenId, double temperature, double alpha)
{
double[] pt = Softmax(teacherLogits, temperature);
double[] ps = Softmax(studentLogits, temperature);
double kl = 0;
for (int i = 0; i < pt.Length; i++)
if (pt[i] > 0) kl += pt[i] * (Math.Log(pt[i]) - Math.Log(ps[i]));
double[] psHard = Softmax(studentLogits, 1.0);
double crossEntropy = -Math.Log(psHard[trueTokenId]);
return alpha * crossEntropy + (1 - alpha) * temperature * temperature * kl; // T² keeps gradient magnitudes comparable across temperatures
}
static double[] Softmax(float[] logits, double t)
{
double max = logits.Max();
double[] exp = logits.Select(z => Math.Exp((z - max) / t)).ToArray();
double sum = exp.Sum();
return exp.Select(e => e / sum).ToArray();
}
When distillation is the right move
HIGH-VOLUME, NARROW TASK Classification, extraction, routing, summarization of a specific document type, structured output — where a frontier model is overkill and cost/latency matter at scale.
LATENCY-CRITICAL A student small enough to meet a tight response-time or on-device constraint.
COST REDUCTION Replace most calls to an expensive model with a cheap specialist, keeping the big model for hard cases (see routing, Section 15).
PRIVACY / OFFLINE Run a distilled open-weight student in your environment.
THE PRACTICAL RECIPE 1) Collect representative inputs (real, anonymized, consented). 2) Have the teacher label/generate outputs, with human spot-checking and filtering for quality. 3) Fine-tune the student
(often with LoRA, Section 11). 4) Evaluate against the teacher AND against ground truth on a held-out set. 5) Deploy behind a router with the teacher as fallback.
Limits and cautions
CAPABILITY CEILING A student rarely exceeds its teacher and usually generalizes worse outside the distribution it was distilled on; broad world knowledge doesn't compress well into few parameters.
ERROR INHERITANCE The student copies the teacher's mistakes and biases — and can amplify them; filter synthetic data and verify against ground truth where possible.
DATA QUALITY Distilled models are only as good as the (synthetic) data: diversity, correctness, and coverage of edge cases matter more than volume.
TERMS OF USE Many proprietary providers' terms restrict using model OUTPUTS to train competing models. Read the license/terms of any teacher whose outputs you use, and check data rights generally.
EVALUATION Measure the student on YOUR task — a small model can look fine on average and fail on the rare, important cases.
10. Fine-Tuning vs. RAG
They change different things
FINE-TUNING Changes the MODEL'S WEIGHTS (or adds trainable adapters). Best at teaching BEHAVIOR: tone, format, task skill, domain conventions, tool-calling style.
RAG Changes the PROMPT by retrieving relevant documents at request time. Best at supplying KNOWLEDGE: facts that are private, changing, or too large to memorize — with citations.
A useful slogan: fine-tuning teaches the model HOW to respond; RAG tells it WHAT to respond with.
The comparison
| Fine-tuning | RAG | |
|---|---|---|
| Primary strength | Behavior, style, format, task specialization | Fresh, private, citable knowledge |
| Knowledge updates | Requires retraining; facts get "baked in" imperfectly | Update the index; effective immediately |
| Citations / grounding | None inherent; can still hallucinate | Natural — answers tied to retrieved sources |
| Access control per user | Impossible to enforce inside weights | Filter retrieval by permissions |
| Deleting data (privacy/legal) | Hard — knowledge diffuses through weights | Delete the document and its chunks |
| Upfront effort | Data collection, training, evaluation, MLOps | Ingestion, chunking, retrieval quality, evaluation |
| Per-request cost | Can be lower (shorter prompts; a smaller specialized model) | Retrieved context adds tokens to every request |
| Latency | Lower (no retrieval step; shorter prompts) | Higher (retrieval + longer prompts) |
| Hallucination on facts | Not reliably reduced; may increase confident wrong facts | Reduced when answers must come from retrieved text |
| Failure modes | Forgetting, overfitting, stale or memorized wrong facts | Retrieval misses, bad chunking, injected documents |
Choose by what you're trying to fix
"IT DOESN'T KNOW OUR DOCS / PRODUCTS / POLICIES" -> RAG.
"ANSWERS MUST BE UP TO DATE OR CITE SOURCES" -> RAG.
"DIFFERENT USERS MAY SEE DIFFERENT DATA" -> RAG with permission filters.
"IT WON'T FOLLOW OUR FORMAT / TONE / HOUSE STYLE" -> prompting first; fine-tune if prompts get long, brittle, or costly.
"WE NEED CONSISTENT STRUCTURED EXTRACTION AT SCALE" -> fine-tune a small model (often via distillation) or use structured outputs with a good prompt.
"A SPECIALIZED CLASSIFICATION / DOMAIN TASK" -> fine-tuning (or a classical/embedding classifier).
"TOO SLOW / TOO EXPENSIVE AT VOLUME" -> distill or fine-tune a smaller model; cache; route.
"IT USES OUR TOOLS BADLY" -> better tool descriptions and few-shot examples first; fine-tune on tool-use traces if needed.
"WE WANT BOTH" -> combine: RAG for facts + a fine-tuned model that is good at using retrieved context and your format (sometimes called RAFT-style training).
The escalation ladder (cheapest and most reversible first)
1. PROMPT ENGINEERING Clear instructions, structure, few-shot examples. (minutes; free; reversible)
2. STRUCTURED OUTPUTS / TOOLS Constrain format; let code do exact work. (hours)
3. RAG Add retrieval for knowledge. (days-weeks; infrastructure)
4. FINE-TUNING (PEFT/LoRA) Adapt behavior on curated examples. (weeks; needs data + evaluation)
5. CONTINUED PRETRAINING Teach a base model a new domain at scale (specialized fields, new languages). (rare; very expensive)
6. TRAIN FROM SCRATCH Almost never justified for an application.
Move down the ladder only when you can show — with an evaluation set — that the previous rung is insufficient.
Types of fine-tuning
FULL FINE-TUNING Update all weights. Strongest adaptation, highest memory/compute cost, a full copy of the model per variant, higher forgetting risk.
PARAMETER-EFFICIENT (PEFT) Train a small number of added or selected parameters (LoRA and relatives, Section 11) while freezing the base. Cheap, fast, swappable. The default for most teams.
SFT Supervised fine-tuning on (prompt, ideal response) pairs.
PREFERENCE TUNING DPO, RLHF-style methods that optimize preferred over rejected responses — to shape style, safety, and judgment.
What good fine-tuning data looks like
QUALITY OVER QUANTITY Hundreds to a few thousand carefully curated, consistent, representative examples often beat tens of thousands of noisy ones. Wrong or inconsistent labels teach the wrong thing.
FORMAT = INFERENCE FORMAT Train with the same chat template, system prompt, and output format you'll use in production.
COVER EDGE CASES Include refusals, "not found" answers, unusual inputs, and hard negatives — not only happy paths.
HOLD OUT A TEST SET Never evaluate on training examples; split by source to avoid leakage.
GUARD PRIVACY AND LICENSES Training data can be memorized and regurgitated; scrub PII and secrets; confirm you have rights to the data and the base model's license allows your use.
DON'T TRAIN FACTS YOU'LL CHANGE Facts that change belong in retrieval, not weights.
Fine-tuning pitfalls
CATASTROPHIC FORGETTING Narrow training erodes general abilities; evaluate general behavior too, and prefer PEFT with modest learning rates and epochs.
OVERFITTING The model memorizes training examples; watch the validation loss and held-out task metrics.
TEACHING HALLUCINATION Training on answers requiring knowledge the model lacks teaches it to answer confidently anyway.
STALE EXPERTISE Weights freeze a snapshot; re-tune on a schedule or pair with RAG.
EVALUATION GAP Without an evaluation set you can't tell whether fine-tuning helped. Build it BEFORE you train.
OPERATIONAL BURDEN Fine-tuned models need versioning, deployment, monitoring, and retraining pipelines (see AI Production / MLOps).
Hosted versus self-managed fine-tuning
HOSTED FINE-TUNING Some providers (including Azure OpenAI for supported models) let you upload training data and get a private fine-tuned deployment: simplest operations, but limited model choice, per-token
training and (often higher) inference pricing, and provider-specific lifecycle. Check current model support and costs.
SELF-MANAGED Fine-tune open-weight models yourself (Hugging Face PEFT/TRL, Axolotl, Unsloth and similar tooling — mostly Python), on GPUs you rent or own, then export and serve the result
(Section 11). Maximum control, with the operational burden that implies. The .NET role is usually orchestration, evaluation, and serving.
11. LoRA and QLoRA
The problem LoRA solves
Full fine-tuning updates every weight: for a 7B model that means storing gradients and optimizer states for 7 billion parameters — several times the model's own memory — and keeping a full 14 GB copy per variant. LoRA (Low-Rank Adaptation) observes that the change a fine-tune makes to a weight matrix tends to be low-rank, and trains only that small change.
How LoRA works
FREEZE the pretrained weight matrix W (d x k).
LEARN a low-rank update: ΔW = B · A with A: (r x k), B: (d x r), rank r ≪ min(d, k)
Effective weight: W' = W + (α / r) · B · A (α is a scaling hyperparameter)
INITIALIZATION: A ~ small random values, B = 0 -> ΔW = 0 at the start, so training begins exactly at the pretrained model.
PARAMETER COUNT (one matrix, d = k = 4,096, rank r = 16):
full matrix: 4,096 x 4,096 = 16,777,216
LoRA (A + B): 16 x (4,096 + 4,096) = 131,072 ≈ 0.78 % of the matrix
WHERE TO APPLY IT Typically the attention projections (query and value at minimum; often all of q, k, v, o) and frequently the MLP's linear layers too. More modules = more capacity, more parameters.
KEY HYPERPARAMETERS rank r (commonly 8-64: higher for harder, more diverse tasks), alpha (often set around r to 2r), dropout (small, e.g. 0.05), learning rate (higher than full fine-tuning, often ~1e-4 to 2e-4),
epochs (usually 1-3), and target modules.
RESULT A tiny "adapter" file (megabytes, not gigabytes) that sits on top of the unchanged base model.
Why it matters operationally
MEMORY & COST Gradients and optimizer states exist only for the small adapter, slashing training memory — fine-tuning big models on modest hardware becomes possible.
SWAPPABLE ADAPTERS One base model, many adapters (per task, customer, language, or tone). Serving stacks can load several adapters on a single base model and select one per request ("multi-LoRA serving"),
which is far cheaper than hosting many full models.
NO INFERENCE PENALTY IF MERGED Fold ΔW into W (W' = W + (α/r)·BA) and ship an ordinary model with zero added latency — or keep adapters separate for flexibility, accepting a small overhead.
LESS FORGETTING The base is frozen, so general capabilities are better preserved (though not guaranteed).
REVERSIBLE Remove the adapter to restore the original model.
QLoRA: LoRA on a quantized base
QLoRA makes fine-tuning even cheaper by loading the frozen base model in 4-bit precision and training LoRA adapters (kept in higher precision) on top, backpropagating through the quantized weights:
4-BIT NORMALFLOAT (NF4) A 4-bit data type designed for normally distributed weights, more accurate than naive 4-bit integers.
DOUBLE QUANTIZATION Quantize the quantization constants too, saving a bit more memory.
PAGED OPTIMIZERS Avoid out-of-memory spikes by paging optimizer state between GPU and CPU.
EFFECT Weights cost ~0.5 byte/parameter instead of 2, so a large model that normally needs a multi-GPU server can be fine-tuned on a single GPU — with quality close to 16-bit LoRA
in the reported results. Training is somewhat slower due to dequantization overhead.
A QLoRA configuration (Python — fine-tuning tooling lives in this ecosystem)
# Illustrative: Hugging Face transformers + peft + bitsandbytes (+ trl for the training loop). Verify current APIs and versions.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
bnb = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=torch.bfloat16,
)
base = AutoModelForCausalLM.from_pretrained("<base-model-id>", quantization_config=bnb, device_map="auto")
base = prepare_model_for_kbit_training(base)
lora = LoraConfig(
r=16, lora_alpha=32, lora_dropout=0.05,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
task_type="CAUSAL_LM",
)
model = get_peft_model(base, lora)
model.print_trainable_parameters() # typically well under 1-2% of all parameters
# ... train with your formatted dataset (use the SAME chat template you'll use at inference), evaluate, then save the adapter
From adapter to a .NET application
1. EVALUATE the adapter on a held-out set against the base model and the prompting/RAG alternatives.
2. MERGE the adapter into the base (optional) and, if desired, QUANTIZE for serving; or keep the adapter separate for multi-LoRA serving.
3. SERVE the result:
- An OpenAI-compatible server (vLLM, Ollama, and similar) — then your .NET code is just another IChatClient (below).
- ONNX Runtime GenAI, after exporting the merged model to ONNX — in-process or local inference from .NET (Section 12).
4. VERSION the base model, adapter, training data version, and hyperparameters; record the version on traces (AI Production / MLOps, Section 2).
5. MONITOR drift and quality like any model; re-tune as needs change.
// A self-hosted open-weight (or LoRA-adapted) model behind an OpenAI-compatible endpoint is consumed like any other chat model
IChatClient chat = new OpenAIClient(
new ApiKeyCredential("not-needed-locally"),
new OpenAIClientOptions { Endpoint = new Uri("http://llm.internal:8000/v1") }) // vLLM / Ollama / similar
.GetChatClient("my-finetuned-model")
.AsIChatClient();
ChatResponse response = await chat.GetResponseAsync("Classify this ticket: 'I was charged twice.'");
Related methods
Variants of LoRA (DoRA, rsLoRA, LoRA+, and others) and other PEFT families (adapters, prefix/prompt tuning, IA3) exist; LoRA/QLoRA remain the most widely used. Preference tuning (DPO) is commonly run
on top of LoRA as well. Treat "which variant" as an experiment to evaluate, not a given.
12. Small Language Models (SLMs)
Smaller, specialized, and surprisingly capable
There is no hard line, but small language models generally means models from a few hundred million to roughly 10-15 billion parameters — small enough to run on a single consumer GPU, a laptop CPU, a phone, or an NPU. Families from several vendors and the open community (for example Microsoft's Phi models, Google's Gemma, smaller Llama, Mistral, and Qwen variants) have shown that with high-quality (often curated or synthetic) training data, good architecture, distillation, and careful post-training, small models can match older, much larger ones on many tasks.
Why SLMs matter
COST A fraction of the per-token (or per-GPU) cost of frontier models.
LATENCY Faster time to first token and tokens per second; viable for real-time, interactive, and high-throughput pipelines.
PRIVACY & RESIDENCY Run entirely inside your boundary, on-device, or offline — data never leaves.
EDGE & OFFLINE Laptops, phones, factory floors, vehicles, retail devices, with NPUs/GPUs/CPUs.
FINE-TUNING Cheap to specialize with LoRA/QLoRA (Section 11), so a tuned SLM can beat a general giant on a narrow task.
SPECULATIVE DECODING A natural "draft" model to accelerate a larger model (Section 6).
COMPONENTS Routers, guard/moderation models, classifiers, extractors, query rewriters, rerankers inside larger systems.
What SLMs are good — and not good — at
GOOD AT Classification, extraction, routing and intent detection, short summarization, structured output, rewriting, FAQ-style Q&A grounded by retrieval, simple tool use, on-device assistants,
and narrow domains after fine-tuning.
WEAKER AT Broad world knowledge (they can't store as many facts — pair with RAG), complex multi-step reasoning, long-context tasks, subtle instruction-following in messy prompts, robustness to
adversarial input, multilingual breadth (varies), and safety behavior (verify; add guardrails).
So: evaluate on YOUR task; use SLMs where the task is narrow and well-defined, and fall back to a larger model when they struggle (Section 15).
Running an SLM from .NET
ONNX RUNTIME GenAI The Microsoft.ML.OnnxRuntimeGenAI package runs generative models exported to ONNX (CPU, CUDA, DirectML, and NPU paths), handling tokenization, the KV cache, and sampling.
OPENAI-COMPATIBLE LOCAL SERVERS Ollama, llama.cpp-based servers, vLLM, and local runtimes such as Microsoft's Foundry Local expose an OpenAI-style API consumed through IChatClient.
LLAMA.CPP BINDINGS .NET bindings (for example LLamaSharp) run GGUF models in-process.
// dotnet add package Microsoft.ML.OnnxRuntimeGenAI (APIs have changed across versions — check the current documentation)
using Microsoft.ML.OnnxRuntimeGenAI;
using var model = new Model(@"C:\models\phi-onnx-int4");
using var tokenizer = new Tokenizer(model);
using var stream = tokenizer.CreateStream();
string prompt = "<|user|>\nExplain dependency injection in one sentence.<|end|>\n<|assistant|>\n"; // the model's chat template matters!
using var generatorParams = new GeneratorParams(model);
generatorParams.SetSearchOption("max_length", 512);
using var generator = new Generator(model, generatorParams);
generator.AppendTokens(tokenizer.Encode(prompt)[0]); // newer API shape; older versions used SetInputSequences + ComputeLogits
while (!generator.IsDone())
{
generator.GenerateNextToken();
Console.Write(stream.Decode(generator.GetSequence(0)[^1])); // stream tokens as they are produced
}
// The package also offers an IChatClient implementation so a local model plugs into the same abstractions as hosted ones
// (check current availability and constructor shape):
IChatClient local = new OnnxRuntimeGenAIChatClient(@"C:\models\phi-onnx-int4");
Practical guidance
PICK A QUANTIZED BUILD 4-bit weights make multi-billion-parameter models practical on CPUs and small GPUs (Section 8); measure quality loss on your task.
MATCH THE CHAT TEMPLATE The wrong role/special-token format is the commonest cause of "the small model is dumb."
CONSTRAIN OUTPUT Use grammar/schema-constrained decoding or validation for structured results — small models drift off-format more often.
COMBINE WITH RAG Let retrieval supply facts; let the SLM do language work.
GUARDRAILS Add input/output moderation and validation; don't assume the safety behavior of a larger model.
BENCHMARK ON TARGET HARDWARE Tokens/sec and memory on the actual CPU/GPU/NPU, at realistic concurrency (see AI Production / MLOps, Section 10).
USE AS A CASCADE TIER Handle the easy majority with the SLM; escalate uncertain or hard requests to a larger model (Section 15).
13. Open-Weight vs. Proprietary Models
Terminology: "open-weight" is not "open source"
PROPRIETARY (CLOSED) Accessed through an API or managed service; weights and training details are not released (many frontier models).
OPEN-WEIGHT The trained WEIGHTS are downloadable, so you can run, fine-tune, and quantize the model yourself. The license varies and may restrict use.
OPEN-SOURCE A stricter notion: permissive license plus (under emerging definitions) openness about training data and code. Many "open" models are open-weight only.
LICENSES MATTER Some open-weight models use permissive licenses (for example Apache-2.0 or MIT); others use custom "community" licenses with conditions — acceptable-use restrictions, attribution,
limits for very large companies, or restrictions on using outputs to train other models. READ THE LICENSE and have legal review it for commercial use.
The trade-offs
| Dimension | Proprietary (hosted) | Open-weight (self-hosted or hosted by a third party) |
|---|---|---|
| Peak capability | Typically the frontier | Often close, sometimes behind on the hardest tasks; narrowing |
| Time to first result | Minutes — an API key | Days — infrastructure, serving, evaluation |
| Cost structure | Pay per token; scales linearly with usage | Fixed GPU/ops cost plus engineering; cheaper at high, steady utilization |
| Control & customization | Prompting, limited fine-tuning options | Full control: fine-tune, quantize, modify, constrain decoding |
| Data privacy / residency | Data goes to the provider (mitigated by terms and private networking) | Data stays in your environment if self-hosted |
| Lock-in & lifecycle | Provider changes, deprecates, or reprices models on its schedule | You hold the weights; no forced retirement (but you own upgrades) |
| Safety & guardrails | Provider safety training and filters included | Your responsibility: moderation, validation, red-teaming |
| Operations | None for the model; you manage quotas | GPUs, scaling, serving stack, monitoring, patching, on-call |
| Reliability / SLA | Provider SLAs; regional capacity and rate limits | Your SLOs; capacity planning is yours |
| Multimodal, tool use, long context | Often the most mature | Improving; check per model |
| Supply-chain risk | Provider trust | Verify artifacts: safe formats (safetensors/ONNX), hashes, provenance (see AI Security, Section 12) |
| Compliance | Provider certifications and contractual terms | Your own compliance posture, with full control of data flows |
A cost framing
HOSTED: monthly cost ≈ tokens per month x price per token (plus any reserved-capacity fees)
SELF-HOSTED: monthly cost ≈ GPU instance cost x hours x replicas + engineering and operations time + storage/networking
BREAK-EVEN: depends almost entirely on UTILIZATION. A GPU that is 15% busy is expensive per token; one at 70% utilization on a quantized, well-batched model can be far cheaper than API pricing.
Include the people cost honestly — and the cost of quality differences (a cheaper model that needs more retries, review, or escalation isn't cheaper).
A practical decision guide
START HOSTED when you're proving value, traffic is modest or spiky, you need top quality fast, and data terms are acceptable.
CONSIDER OPEN-WEIGHT when: data cannot leave your boundary; volume is high and steady enough to keep GPUs busy; you need deep customization (fine-tuning, constrained decoding, adapters); you need offline/edge
operation; you want to avoid dependency on a provider's pricing and deprecations; or a small tuned model meets the quality bar far more cheaply.
MIX: route by task and sensitivity — a self-hosted SLM for classification, routing, and PII-bearing requests; a hosted frontier model for hard reasoning and generation (Section 15).
Cloud model catalogs (such as Azure AI Foundry's) host both kinds, letting you consume open models as managed endpoints without running GPUs yourself.
ABSTRACT the model behind IChatClient so the choice is reversible (see Microsoft.Extensions.AI), and keep an evaluation set so you can compare models on YOUR task.
14. Multimodal LLMs
Models that read more than text
Multimodal LLMs accept (and sometimes produce) several modalities — text, images, audio, video, documents — in one conversation. Instead of separate OCR, speech-to-text, and vision pipelines glued to a text model, one model reasons over everything together.
How images become tokens
TYPICAL ARCHITECTURE (vision-language models)
IMAGE ──> VISION ENCODER (a Vision Transformer, ViT: splits the image into patches, e.g. 14x14 pixels, and embeds each)
──> PROJECTOR / ADAPTER (maps image-patch embeddings into the language model's embedding space)
──> the resulting "image tokens" are placed in the sequence alongside text tokens
The LLM then attends over the combined sequence exactly as it does over text.
DESIGN VARIANTS
EARLY FUSION / UNIFIED Image tokens go directly into the decoder's input sequence (common today).
CROSS-ATTENTION The text decoder cross-attends to image features in dedicated layers (Flamingo-style).
NATIVE MULTIMODAL Trained from the start on interleaved text, image, and audio data (some frontier models), rather than bolting a vision encoder onto a text model.
AUDIO Audio encoders (Whisper-style) feed speech into the model, or natively multimodal models take audio directly and may emit speech.
VIDEO Sampled frames (plus audio/time information) treated as sequences of images — expensive in tokens.
OUTPUT Many "multimodal" models only OUTPUT text; image or audio generation is often handled by separate generative models (diffusion models for images) or a model with native generation.
Token cost of images
An image consumes tokens in proportion to its size/detail — a high-resolution image can cost hundreds to thousands of tokens (providers often tile images and charge per tile, with a low-detail mode that
uses a small fixed cost). Video and multi-page documents multiply this. So image-heavy workloads dominate input cost and fill the context window quickly. Check your provider's image-token accounting, downscale
images to the smallest size that preserves what the task needs, and cache or pre-process repeated images.
Using multimodal models from .NET
using Microsoft.Extensions.AI;
public record InvoiceFields(string Vendor, string InvoiceNumber, string Date, decimal Total);
var message = new ChatMessage(ChatRole.User,
[
new TextContent("Extract the vendor, invoice number, date, and total from this invoice."),
new DataContent(imageBytes, "image/jpeg"), // or: new UriContent(new Uri("https://..."), "image/jpeg")
]);
ChatResponse<InvoiceFields> response = await chat.GetResponseAsync<InvoiceFields>([message]);
InvoiceFields invoice = response.Result; // shape is guaranteed; VALIDATE the values (does the total equal the sum of the lines?)
Strengths, limits, and how to engineer around them
STRENGTHS Flexible zero-shot understanding of screenshots, charts, diagrams, handwriting, receipts, UI, and photos; question answering and reasoning that combines text and image; messy real-world documents.
LIMITS
FINE DETAIL & SMALL TEXT Downscaled images lose small print; crop and zoom to regions that matter.
COUNTING & SPATIAL REASONING Counting many objects and precise positions/boxes are unreliable; use a dedicated detector (see Computer Vision with .NET) and let the LLM interpret.
HALLUCINATION Models can invent text or details not in the image — especially in OCR-like extraction. Validate against business rules and, where possible, a classical OCR cross-check.
CHARTS & TABLES Reading exact values from charts and complex tables can be wrong; prefer structured sources or layout-aware extraction (Document Intelligence) when precision matters.
CONTEXT & COST Many images exhaust the window and the budget.
LATENCY Image-heavy prompts increase prefill time (TTFT).
PRIVACY Images may contain faces, IDs, medical or financial data — control where they go (see AI Security and Computer Vision with .NET).
PROMPT INJECTION Text inside an image can carry instructions ("visual prompt injection"); treat image content as untrusted data.
VARIABILITY Support for formats, sizes, number of images, and modalities differs by model; check limits.
PATTERNS
HYBRID PIPELINES Classical OCR/detection for precision and speed + the LLM for interpretation and normalization; send the LLM only the crops or fields that need judgment.
DOCUMENTS For PDFs, extract text where it exists and send page images only for scanned or visually complex pages.
EVALUATE Build a labeled set from real images, measure field-level accuracy, hallucination rate, and cost per image; route cheap/easy images to a smaller model.
15. Model Routing and Fallback Strategies
One model for everything is the expensive default
Different requests need different models: a short classification doesn't need a frontier model, an image needs a vision-capable one, a document with personal data may need a private model, and any model can be rate-limited or down. Routing chooses the model per request; fallback keeps the system working when the chosen model fails. Together they cut cost and latency, improve availability, and let you use the best model for each job (see also LLM Application Development, Section 9, and AI Production / MLOps, Section 9).
Routing strategies
STATIC / RULE-BASED Route by feature or endpoint ("ticket classification -> small model; contract analysis -> strong model"). Simple, predictable, testable. The right starting point.
CAPABILITY-BASED Route by what the request NEEDS: images -> a vision model; very long input -> a long-context model; tool use or structured output -> one that supports it; a specific language -> one strong in it.
SENSITIVITY-BASED Route by data classification: requests containing PII/confidential data go to a self-hosted or private-network model; others may use a hosted frontier model.
COMPLEXITY / HEURISTIC Use signals — input length, presence of code or math, number of constraints, conversation depth, tools requested — to guess difficulty.
CLASSIFIER / LEARNED ROUTER A small model (an embedding + classifier, or a fine-tuned SLM) predicts whether the cheap model will suffice, trained on labeled examples of "strong model needed vs. not."
Research and open-source routers trained on preference data exist; evaluate before trusting.
LLM-AS-ROUTER Ask a small LLM to triage the request. Flexible, but adds latency and cost and is itself non-deterministic — use sparingly.
CASCADE (TRY CHEAP, VERIFY, ESCALATE) Send to the cheap model first; check the result with validators, confidence signals, or a judge; escalate to a stronger model only on failure. Most cost savings,
at the price of extra latency on escalated requests and wasted spend on failed cheap attempts.
CONFIDENCE-BASED Use signals such as token log-probabilities (where available), agreement across multiple samples (self-consistency), schema/rule validation, or refusal/"unsure" markers to decide
whether to escalate.
COST- AND BUDGET-AWARE Factor in tenant tier, remaining budget, and current load: premium tenants get the strong model; free tier gets the small one; degrade gracefully when a budget is nearly spent.
LATENCY-AWARE Pick the faster model when the user is waiting and the task allows it.
ADAPTIVE (BANDIT / A-B) Learn from outcomes (feedback, task success, cost) which model works best per request type, exploring a little while exploiting what works.
A router abstraction and a heuristic implementation
public sealed record RouteDecision(string Primary, IReadOnlyList<string> Fallbacks, string Reason);
public sealed record RouteContext(string Feature, string TenantTier, bool ContainsSensitiveData);
public interface IModelRouter
{
RouteDecision Decide(IReadOnlyList<ChatMessage> messages, ChatOptions? options, RouteContext context);
}
public sealed class HeuristicRouter(Tokenizer tokenizer) : IModelRouter
{
public RouteDecision Decide(IReadOnlyList<ChatMessage> messages, ChatOptions? options, RouteContext context)
{
// 1. Sensitivity first: data classification can't be traded for cost
if (context.ContainsSensitiveData)
return new("private-slm", ["private-large"], "sensitive data stays in-boundary");
// 2. Capability requirements
bool hasImage = messages.SelectMany(m => m.Contents).OfType<DataContent>().Any(d => d.MediaType?.StartsWith("image/") == true);
if (hasImage)
return new("hosted-vision", ["hosted-vision-alt"], "image input");
int inputTokens = messages.Sum(m => tokenizer.CountTokens(m.Text ?? ""));
if (inputTokens > 100_000)
return new("hosted-long-context", ["hosted-strong"], "very long input");
// 3. Difficulty and tier
bool usesTools = options?.Tools is { Count: > 0 };
if (usesTools || context.Feature is "contract-analysis" or "code-review")
return new("hosted-strong", ["hosted-mid"], "tool use or hard task");
if (context.TenantTier == "free")
return new("private-slm", ["hosted-mid"], "free tier: cheapest adequate model");
// 4. Default: a cheap model, with escalation available
return new("hosted-small", ["hosted-mid", "hosted-strong"], "default");
}
}
A routing and fallback chat client (with circuit breakers)
public sealed class RoutingChatClient(
IReadOnlyDictionary<string, IChatClient> clients,
IModelRouter router,
ICircuitBreakers breakers,
IRouteContextAccessor contextAccessor,
ILogger<RoutingChatClient> logger) : IChatClient
{
public async Task<ChatResponse> GetResponseAsync(
IEnumerable<ChatMessage> messages, ChatOptions? options = null, CancellationToken cancellationToken = default)
{
var list = messages.ToList();
RouteDecision decision = router.Decide(list, options, contextAccessor.Current);
Exception? lastError = null;
foreach (string name in new[] { decision.Primary }.Concat(decision.Fallbacks))
{
if (breakers.IsOpen(name)) continue; // skip models that are currently failing
try
{
ChatResponse response = await clients[name].GetResponseAsync(list, options, cancellationToken);
breakers.RecordSuccess(name);
logger.LogInformation("Routed to {Model}: {Reason}", name, decision.Reason); // always log the decision for cost and quality analysis
return response;
}
catch (Exception ex) when (IsFailoverCandidate(ex) && !cancellationToken.IsCancellationRequested)
{
breakers.RecordFailure(name);
lastError = ex;
logger.LogWarning(ex, "Model {Model} failed; trying next fallback", name);
}
}
throw new InvalidOperationException("All routes failed or are unavailable.", lastError);
}
public async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
IEnumerable<ChatMessage> messages, ChatOptions? options = null,
[EnumeratorCancellation] CancellationToken cancellationToken = default)
{
var list = messages.ToList();
RouteDecision decision = router.Decide(list, options, contextAccessor.Current);
Exception? lastError = null;
foreach (string name in new[] { decision.Primary }.Concat(decision.Fallbacks))
{
if (breakers.IsOpen(name)) continue;
await using var enumerator = clients[name].GetStreamingResponseAsync(list, options, cancellationToken).GetAsyncEnumerator(cancellationToken);
ChatResponseUpdate first;
// Fail over ONLY BEFORE the first token reaches the caller. After output has started, switching models mid-answer would produce an incoherent response.
try
{
if (!await enumerator.MoveNextAsync()) yield break;
first = enumerator.Current;
breakers.RecordSuccess(name);
}
catch (Exception ex) when (IsFailoverCandidate(ex) && !cancellationToken.IsCancellationRequested)
{
breakers.RecordFailure(name);
lastError = ex;
continue;
}
yield return first;
while (await enumerator.MoveNextAsync())
yield return enumerator.Current; // errors after this point propagate to the caller
yield break;
}
throw new InvalidOperationException("All routes failed or are unavailable.", lastError);
}
// Transient/provider-side failures justify failover; client bugs (bad request) do not — they'd fail on every model
private static bool IsFailoverCandidate(Exception ex) => ex switch
{
HttpRequestException or TimeoutException => true,
ClientResultException c => c.Status is 408 or 429 or >= 500,
_ => false
};
public object? GetService(Type serviceType, object? serviceKey = null) => null;
public void Dispose() { foreach (IChatClient c in clients.Values) c.Dispose(); }
}
A cascade with verification
// Try the cheap model; accept its answer only if it passes validation; otherwise escalate
public async Task<Classification> ClassifyWithCascadeAsync(string text, CancellationToken ct)
{
ChatResponse<Classification> cheap = await _small.GetResponseAsync<Classification>(BuildMessages(text), new ChatOptions { Temperature = 0 }, cancellationToken: ct);
if (_allowedLabels.Contains(cheap.Result.Category) && cheap.Result.Confidence >= 0.8)
{
_metrics.RecordRoute("small", escalated: false);
return cheap.Result;
}
ChatResponse<Classification> strong = await _strong.GetResponseAsync<Classification>(BuildMessages(text), new ChatOptions { Temperature = 0 }, cancellationToken: ct);
_metrics.RecordRoute("strong", escalated: true);
return strong.Result;
}
Fallback strategies
ERROR-BASED FAILOVER 429 / 5xx / timeout on the primary -> the same model in another region/deployment, then an equivalent model from another provider, then a smaller model. Honor Retry-After; use
jittered backoff; trip a CIRCUIT BREAKER after repeated failures so you stop sending traffic to a failing backend and probe it periodically (half-open) until it recovers.
QUALITY-BASED ESCALATION Output fails schema/business validation, a citation check, or a judge threshold -> retry once with a stronger model (the cascade above).
CAPACITY-BASED SPILLOVER Reserved/provisioned capacity exhausted -> spill into pay-as-you-go capacity or a different deployment.
GRACEFUL DEGRADATION When everything is down or over budget: serve a cached answer, a simpler non-LLM result (keyword search, a template), or an honest "temporarily unavailable" — rather than failing hard.
HEDGED REQUESTS Send the same request to two backends and use whichever answers first, cancelling the other — cuts tail latency at roughly double cost. Reserve for latency-critical, low-volume paths.
MODEL ALIASES Application code requests a LOGICAL name ("support-chat"); configuration maps it to concrete models and fallbacks, so changes need no code deploy.
The hard parts of routing and fallback
PROMPT PORTABILITY Prompts tuned for one model may behave differently on another (instruction following, formatting, refusals, tool-call syntax). Test each fallback with the SAME evaluation suite;
keep model-specific prompt variants when needed. A fallback that returns malformed output isn't a fallback.
FEATURE PARITY Fallback models may lack tools, vision, structured outputs, or long context; the router must only choose models that satisfy the request's requirements.
CONSISTENCY WITHIN A CONVERSATION Switching models mid-conversation can change tone or contradict earlier turns; consider pinning a conversation to a model unless it fails.
MID-STREAM FAILURES Once tokens have streamed, you can't cleanly switch (as in the code above); decide whether to show a partial answer with an error, or retry fully behind a non-streaming path.
THE ROUTER'S OWN COST Routing logic, classifiers, and judge calls add latency and spend; the savings must clearly exceed them.
EVALUATING ROUTING Measure the COST-QUALITY CURVE: for a labeled set, compute quality and cost for "always small," "always strong," your router, and an oracle router (best model per request). A good router
sits near the oracle: close to strong-model quality at a fraction of the cost. Track the share routed to each model and the escalation rate in production.
DRIFT Request mix, model versions, and prices change — re-evaluate routing rules regularly.
SECURITY Never let the user (or injected text) choose the model or tier; derive it from server-side policy and entitlements (see AI Security, Section 12). Route sensitive data only to approved models.
OBSERVABILITY Log every routing decision (chosen model, reason, fallbacks tried, escalation, cost, latency, outcome) and tag traces with it (see AI Observability).
16. Back-of-the-Envelope Calculators
Numbers drive architecture decisions: Will it fit? How many users per GPU? What does 4-bit save? How fast can it decode? A few formulas answer most of those questions.
public static class LlmMath
{
private const double GiB = 1024.0 * 1024 * 1024;
// Approximate parameter count of a dense decoder-only transformer: attention (4d²) + MLP (2 * m * d²) per layer, plus embeddings.
// (Gated MLPs, grouped-query attention, and tied embeddings change the details; this is for sizing, not accounting.)
public static long ApproxParameters(int layers, int dModel, int vocabSize, int mlpMultiplier = 4) =>
(long)layers * (4L * dModel * dModel + 2L * mlpMultiplier * dModel * dModel) + (long)vocabSize * dModel;
// Memory for the weights alone
public static double WeightGiB(long parameters, double bytesPerParameter) => parameters * bytesPerParameter / GiB;
// KV cache: 2 (K and V) x layers x kvHeads x headDim x tokens x batch x bytes per element
public static double KvCacheGiB(int layers, int kvHeads, int headDim, int tokens, int batch, double bytesPerElement) =>
2.0 * layers * kvHeads * headDim * tokens * batch * bytesPerElement / GiB;
// Trainable parameters LoRA adds for one (d x k) matrix at rank r
public static long LoraParameters(int d, int k, int rank) => (long)rank * (d + k);
// Upper bound on single-stream decode speed: every generated token must read all weights from memory once
public static double DecodeTokensPerSecondUpperBound(double weightBytes, double memoryBandwidthBytesPerSecond) =>
memoryBandwidthBytesPerSecond / weightBytes;
}
EXAMPLES (illustrative)
7B-parameter model weights: FP16 ≈ 13 GiB | INT8 ≈ 6.5 GiB | 4-bit ≈ 3.3 GiB
KV cache, 7B-class with full multi-head attention (32 layers, 32 KV heads, head_dim 128, FP16):
4,096 tokens x 1 sequence = 2.0 GiB x 16 concurrent sequences = 32 GiB
Same with grouped-query attention (8 KV heads): 4,096 tokens x 1 sequence = 0.5 GiB x 16 = 8 GiB
Same with an FP8 KV cache: halve again.
70B-class GQA model (80 layers, 8 KV heads, head_dim 128, FP16): 8,192 tokens x 1 sequence ≈ 2.5 GiB
LoRA, one 4,096 x 4,096 matrix, rank 16: 131,072 trainable parameters vs 16,777,216 (≈ 0.78 %)
DECODE SPEED BOUND: a 7B FP16 model reads ~14 GB per token; on a GPU with ~1 TB/s memory bandwidth, single-stream decoding is capped near 70 tokens/s.
Quantize to 4-bit (≈3.5 GB per token) and the bound rises roughly 4x. Batching amortizes the weight reads across many sequences, which is why serving throughput scales far beyond the single-stream figure.
TTFT is dominated by PREFILL (compute-bound, ~linear in prompt length plus attention's quadratic term); DECODE speed is bandwidth-bound — optimize them differently.
SIZING CHECKLIST FOR SELF-HOSTING
GPU memory ≥ weights (at chosen precision) + KV cache x concurrent sequences + activations/overhead (≈10-20%)
Throughput ≈ decode bound x effective batch size (until compute or cache memory saturates)
Then LOAD TEST: the formulas find the neighborhood; measurement finds the answer (see AI Production / MLOps, Section 9).
17. Interview-Style Questions
Concise model answers — expand with examples from your own experience.
1. Explain self-attention and why it's scaled by √d_k.
Each token is projected to a query, key, and value. Attention scores are query-key dot products, softmaxed into weights, and used to take a weighted sum of values, so each token gathers relevant context from others. Without dividing by √d_k, dot products grow with dimension, pushing the softmax into saturation (near one-hot) and giving tiny gradients; scaling keeps the variance stable.
2. Why is attention quadratic, and how do modern systems cope?
The score matrix is n × n, so compute and (naively) memory grow with the square of sequence length. FlashAttention computes exact attention without materializing the full matrix and with far less memory traffic; GQA/MQA shrink the KV cache; sliding-window and sparse patterns cut cost; prefix caching avoids recomputing shared prompts.
3. Why does a transformer need positional encoding? What is RoPE?
Attention is permutation-invariant, so order must be injected. RoPE rotates query and key vectors by position-dependent angles so their dot product depends on relative distance, which generalizes well and is used by most modern open models; context extension methods (interpolation, NTK/YaRN scaling) rescale the rotations.
4. What is the KV cache and why does it dominate serving memory?
During decoding, each layer's keys and values for all previous tokens are cached so a new token only computes its own K/V. The cache grows linearly with sequence length and batch size (2 × layers × kv_heads × head_dim × tokens × bytes), so with many concurrent users it often exceeds the weights' size. GQA, KV quantization, PagedAttention, and prefix caching manage it.
5. Prefill versus decode?
Prefill processes the prompt in parallel and is compute-bound, determining time to first token. Decode generates one token per step and is memory-bandwidth-bound, determining tokens per second. They are optimized differently (chunked prefill, continuous batching, quantization).
6. How does temperature differ from top-p?
Temperature rescales the whole distribution (sharper or flatter); top-p truncates to the smallest set of tokens whose cumulative probability reaches p, adapting to how confident the model is. Top-k keeps a fixed number of tokens. Usually tune temperature or top-p, not both.
7. Why isn't temperature 0 perfectly deterministic?
Greedy decoding is deterministic in principle, but floating-point nondeterminism from parallel reductions, batching, kernel selection, and hardware differences can flip near-ties; some providers also update models. Design for approximate repeatability.
8. How does BPE work, and what are the practical consequences of tokenization?
BPE starts from bytes/characters and repeatedly merges the most frequent adjacent pairs up to a vocabulary size. Consequences: cost and context limits are in tokens; non-English text and code often tokenize less efficiently; character- and digit-level tasks are hard because the model sees tokens; whitespace and chat-template details matter.
9. What is quantization, and how do you decide how far to go?
Storing weights (and optionally activations/KV cache) in fewer bits to cut memory and bandwidth. Weight-only 4-bit is common for inference. Start from higher precision, step down while measuring task-specific quality (not just perplexity), and compare cost per inference; larger models tolerate quantization better than small ones, and outliers are the main difficulty.
10. Fine-tuning or RAG for a company knowledge assistant?
RAG: knowledge is private, changing, needs citations, per-user permissions, and deletion. Fine-tuning teaches behavior and format, not reliably facts. Often combine: RAG for knowledge, a (LoRA-)tuned model for style/tool use, after prompt engineering shows its limits and with an evaluation set.
11. Explain LoRA and why QLoRA is cheaper.
LoRA freezes the base weights and learns low-rank updates ΔW = BA (rank r ≪ d), training a tiny fraction of parameters and producing a small, swappable adapter that can be merged for zero inference overhead. QLoRA additionally keeps the frozen base in 4-bit NF4 (with double quantization and paged optimizers) so large models fine-tune on a single GPU with little quality loss.
12. How does distillation differ from fine-tuning?
Distillation is training a smaller student to mimic a teacher — via soft targets or, for LLMs, usually teacher-generated synthetic data — to compress capability. Fine-tuning adapts a model to a task with labeled data. They combine: distill teacher outputs into a small model using LoRA. Watch capability ceilings, inherited errors, and provider terms restricting use of outputs.
13. When would you choose a small language model?
Narrow, high-volume, latency- or privacy-sensitive tasks (classification, extraction, routing, on-device) — ideally fine-tuned, paired with RAG for facts, and backed by a larger-model fallback for hard cases.
14. Open-weight or proprietary?
Proprietary for fastest start and top capability with no ops; open-weight for data control, deep customization, offline/edge use, predictable high-utilization cost, and no forced deprecations — at the price of operating GPUs, safety guardrails, and license review. Abstract behind IChatClient and evaluate on your task; many systems mix both.
15. Design a model router with fallbacks.
Rules by capability and data sensitivity first (vision, long context, PII → private model), then difficulty/tier heuristics or a small classifier, with a cascade that validates cheap-model output and escalates. Fallbacks: regional/provider failover with circuit breakers and jittered retries, quality-based escalation, graceful degradation. Evaluate on a cost-quality curve versus an oracle, keep prompts portable across models, fail over only before streaming starts, never let users pick the model, and log every decision.
16. How do multimodal LLMs process images, and what are the pitfalls?
A vision encoder turns image patches into embeddings, a projector maps them into the LLM's space, and they enter the sequence as tokens. Pitfalls: high token cost, weak counting/localization and fine-detail reading, hallucinated extraction, visual prompt injection, and privacy — mitigate with hybrid OCR/detection pipelines, validation, cropping, and routing.
17. A 7B model fits in 14 GB, yet your GPU runs out of memory at 20 concurrent users. Why?
KV cache. At 0.5 MiB per token (full MHA, FP16) a 4k-token conversation is ~2 GiB; 20 of them need ~40 GiB. Use GQA models, quantize the cache, enable PagedAttention/prefix caching, cap context and concurrency, or scale out.
18. Common Pitfalls
| Pitfall | Why it hurts | Better approach |
|---|---|---|
| Treating advertised context length as usable quality | Quality degrades with distraction and position; costs scale | Measure your task at real context lengths; pack and compact context (Section 5) |
| Putting variable content (timestamps, IDs) at the start of the prompt | Defeats prefix/KV caching; higher cost and TTFT | Static content first, variable content last (Sections 5, 6) |
| Ignoring the KV cache when sizing GPUs | Out-of-memory or collapsed concurrency despite weights fitting | Size for weights + cache x concurrency; use GQA/FP8 cache/PagedAttention (Section 6) |
| Using the wrong tokenizer or chat template | Silent quality collapse | Pin tokenizer and template with the model; test (Section 4) |
| Counting tokens by word count | Budgets and costs are wrong, especially for non-English and code | Use the real tokenizer; test your languages (Section 4) |
| Asking the model to count letters or do exact arithmetic | Token-level blindness | Give it a tool (Section 4) |
| Changing temperature and top-p together | Hard-to-predict behavior | Tune one at a time; evaluate on a test set (Section 7) |
| Expecting temperature 0 to be perfectly reproducible | Floating-point and batching variation; provider updates | Design for approximate repeatability; repeat evaluation runs (Section 7) |
| Quantizing without measuring task quality | Perplexity can look fine while your task degrades | Evaluate on your own tasks, including edge cases (Section 8) |
| Distilling from a provider whose terms forbid it | Legal and contractual exposure | Check terms of the teacher; prefer permissive teachers or human data (Section 9) |
| Fine-tuning to teach changing facts | Facts go stale, can't be deleted, may be hallucinated | Use RAG for knowledge; fine-tune for behavior (Section 10) |
| Fine-tuning without an evaluation set | Can't tell if it helped or hurt | Build the evaluation set first; compare to prompting and RAG (Section 10) |
| Training and serving with different chat templates | The tuned behavior doesn't appear | Train with the exact inference format (Sections 10, 11) |
| Assuming a small model has frontier-model safety and robustness | Weaker guardrails and instruction following | Add moderation/validation; fall back for hard cases (Section 12) |
| Ignoring open-weight licenses | Commercial or usage restrictions violated | Legal review of license and output terms (Section 13) |
| Comparing self-hosted and hosted cost without utilization | Idle GPUs make self-hosting look cheap on paper and expensive in practice | Model cost at realistic utilization plus engineering time (Section 13) |
| Sending large images without downscaling or cropping | Huge token cost, slow TTFT, lost small text | Right-size and crop images; route to appropriate models (Section 14) |
| Trusting multimodal extraction without validation | Plausible hallucinated values | Business-rule validation and classical OCR cross-checks (Section 14) |
| Letting user input choose the model | Cost abuse and safety bypass | Server-side routing policy (Section 15) |
| Fallback models that were never tested | Malformed output and broken tools during an outage | Evaluate every route with the same suite; keep prompt variants (Section 15) |
| Failing over mid-stream | Incoherent mixed answers | Fail over only before the first token; otherwise surface the error (Section 15) |
| Routing with no measurement | Savings and quality unknown; router drift | Cost-quality curves vs. oracle; log decisions; re-evaluate periodically (Section 15) |
Quick Reference Table
| Topic | Key idea | Practical takeaway |
|---|---|---|
| Transformer | Stacked blocks of self-attention + feed-forward, residuals, normalization; next-token prediction | Cost scales with parameters (memory) and tokens (compute) |
| Self-attention | softmax(QKᵀ/√d)V with a causal mask | Quadratic in sequence length — the root of long-context cost |
| Multi-head / GQA | Parallel attention heads; shared KV heads in GQA/MQA | GQA shrinks the KV cache with little quality loss |
| FlashAttention | Exact attention with less memory traffic | Standard speedup in training and serving |
| Positional encoding | Inject order; RoPE encodes relative position | Context extension needs scaling/long-context training; usable context < headline |
| Tokenization | BPE / WordPiece / Unigram (SentencePiece) subwords | Tokens drive cost and quirks; use the matching tokenizer and chat template |
| Context management | Budget, rank/pack, compact, cache prefix | Static first, variable last; measure quality vs. length |
| KV cache | Stores K/V for past tokens; 2·L·kv_heads·d_head·tokens·batch·bytes | Often the concurrency limit; prefix caching saves cost and TTFT |
| Prefill vs. decode | Compute-bound vs. bandwidth-bound | Optimize TTFT and tokens/sec separately |
| Sampling | Temperature, top-k, top-p, penalties, constrained decoding | Tune one knob; use structured outputs for formats; 0 isn't bit-exact |
| Quantization | Fewer bits per weight/activation/KV | 4-bit weight-only is the common sweet spot; always measure on your task |
| Distillation | Student mimics teacher (soft targets or synthetic data) | Great for narrow high-volume tasks; check terms and ceilings |
| Fine-tuning vs. RAG | Behavior vs. knowledge | Prompt → tools → RAG → fine-tune; combine when needed |
| LoRA / QLoRA | Low-rank adapters; 4-bit frozen base | Cheap, swappable, mergeable; evaluate vs. alternatives |
| SLMs | ~0.5–15B parameter models | Local, private, fast; fine-tune, add RAG, keep a fallback |
| Open-weight vs. proprietary | Control and privacy vs. capability and convenience | Decide on utilization, data rules, licenses; abstract behind IChatClient |
| Multimodal | Vision encoder + projector → image tokens | Costly tokens; use detectors/OCR for precision; validate outputs |
| Routing & fallback | Right model per request; survive failures | Rules → cascades; circuit breakers; fail over before streaming; log decisions |
Conclusion
Large language models look magical from the API, but they are engineered systems with knowable parts: a tokenizer that turns text into subword IDs, embeddings plus a positional scheme that tell the model what and where, stacked transformer blocks whose attention mixes information across tokens, a KV cache that makes token-by-token generation affordable, and a sampler that turns logits into text. Each of those has a direct engineering consequence. Tokenization sets your costs and explains the strange failures (letter counting, arithmetic, multilingual overhead). Attention's quadratic cost and the position scheme explain why long contexts are slow, expensive, and uneven in quality. The KV cache explains prefix-caching discounts, why static content belongs first in a prompt, and why memory — not FLOPs — often decides how many users a GPU can serve. Sampling explains why temperature 0 still wobbles and why structured outputs beat hoping for valid JSON.
The second half of the toolbox is about adapting and arranging models rather than building them. Quantization, distillation, and LoRA/QLoRA let you shrink, specialize, and cheaply customize models, and small language models make private, on-device, and high-volume uses practical. The discipline is to climb the ladder only as far as evidence demands — prompt, then tools and structured outputs, then RAG for knowledge, then fine-tuning for behavior — and to evaluate every step on your own task, because a quantized or distilled model that looks fine on average can fail on the cases that matter. Choosing between open-weight and proprietary models is a decision about data, utilization, licenses, and operational appetite, best kept reversible behind IChatClient. And the way to harvest all of it is routing and fallback: rules for capability and sensitivity, cascades that validate cheap answers and escalate, circuit breakers and tested fallbacks for resilience, honest cost-quality evaluation against an oracle, and a log of every decision. Know the mechanics well enough to do the arithmetic — weights, KV cache, decode bandwidth, LoRA parameters — and you can make architecture decisions with numbers instead of folklore, which is what separates building with LLMs from engineering them.
Found this useful? Feel free to star the repo, open an issue with corrections, or share the "the weights fit on the GPU, but the KV cache didn't" story that made the case for doing the memory math before the load test click better than any architecture diagram.
Top comments (0)