When you stream a response from a Large Language Model and watch words appear one by one, your GPU is operating under a bizarre physical reality: it spends roughly 98% of its time waiting for memory to transfer and less than 2% doing actual math.
If you paste a 2,000-word prompt into an LLM, the model digests the entire prompt almost instantaneously (the prefill phase). But when it generates the 2,000-word answer, it outputs tokens at a steady, sequential drip (the decode phase).
Why does this asymmetry exist? What actually happens under the hood between the moment token $N$ is chosen and the moment token $N+1$ is emitted?
Let us walk through the exact journey of a single token generation step through modern decoder-only transformers (like LLaMA 3 or Mistral), from raw integer IDs to tensor operations, KV caching, logit distributions, and the memory-bandwidth wall.
The Core Asymmetry: Prefill vs. Decode
Before looking at individual matrix multiplications, we must understand the two distinct operational modes of an LLM:
- Prefill (Compute-Bound): When you send a prompt, the model processes all $T$ prompt tokens in parallel. A matrix of dimensions $[T \times d_{\text{model}}]$ multiplies against model weights. The GPU tensor cores stay fully saturated because each weight loaded from VRAM is reused across all $T$ tokens.
- Decode (Memory-Bandwidth Bound): When generating responses, autoregression forces the model to generate one token at a time. The input is a vector of shape $[1 \times d_{\text{model}}]$. To compute this single vector forward pass, the GPU must fetch every single weight parameter (all 8 billion or 70 billion parameters) from VRAM into the compute cores for just two floating-point operations per parameter.
Prefill Phase (Prompt Ingestion):
[Token 1, Token 2, ... Token 2048] ──> [Matrix × Matrix] ──> GPU Tensor Cores 100% Saturated
Decode Phase (Token by Token Generation):
[Token N] ──> [Vector × Matrix] ──> GPU Waiting on Memory Bus (98% Idle Compute)
Now let us follow what happens during one single decode iteration.
Step 1: Input Vector & Rotary Position Embedding (RoPE)
Suppose the model has just emitted the token " lazy" (integer ID 16005 in LLaMA 3's vocabulary).
1. Embedding Lookup
The token integer 16005 acts as a row index into the embedding matrix $W_e \in \mathbb{R}^{V \times d_{\text{model}}}$.
For an 8-billion parameter model (such as LLaMA 3 8B), $d_{\text{model}} = 4096$ and vocabulary size $V = 128,256$.
Retrieving row 16005 yields our starting hidden state vector:
$$x_0 \in \mathbb{R}^{1 \times 4096}$$
# Conceptual PyTorch equivalent
token_id = 16005
x = embedding_table[token_id] # Shape: [1, 4096]
2. Positional Encoding via RoPE
Older architectures (like original GPT-2) added fixed positional vectors to the embedding: $x_0 = x_{\text{token}} + x_{\text{pos}}$.
Modern architectures use Rotary Position Embedding (RoPE). Instead of adding a vector at the bottom, RoPE rotates 2D pairs of features inside the Query ($Q$) and Key ($K$) vectors by an angle proportional to the token's absolute position $m$ in the context window. This allows the dot product $Q K^T$ to naturally encode the relative distance between any two tokens.
Step 2: The Transformer Stack (Repeated Across 32 Layers)
Our 4096-dimensional vector $x$ now travels through a stack of 32 identical transformer layers. Each layer performs two major operations: Grouped-Query Attention and a SwiGLU Feed-Forward Network.
Input Vector x [1, 4096]
│
▼
┌──────────────────────────────────────┐
│ 1. RMSNorm │
│ 2. Q, K, V Projections │
│ 3. RoPE Rotation on Q and K │
│ 4. Read/Write KV Cache in VRAM │
│ 5. Scaled Dot-Product Attention │
│ 6. Output Projection + Residual Add │
├──────────────────────────────────────┤
│ 7. RMSNorm │
│ 8. SwiGLU MLP (Gate, Up, Down Proj) │
│ 9. Residual Add │
└──────────────────────────────────────┘
│ (Repeated 32 times)
▼
Final Hidden State [1, 4096]
Let us look at each sub-step inside a single layer:
A. Pre-Normalization: RMSNorm
Before feeding $x$ into attention, modern models normalize the vector using Root Mean Square Normalization (RMSNorm), which is computationally lighter than LayerNorm because it skips mean-centering:
$$\text{RMS}(x) = \sqrt{\frac{1}{d}\sum_{i=1}^d x_i^2 + \epsilon}, \quad x_{\text{norm}} = \frac{x}{\text{RMS}(x)} \odot \gamma$$
Where $\gamma$ is a learned scaling vector of size 4096.
B. Grouped-Query Attention (GQA) & The KV Cache
In standard Multi-Head Attention (MHA), if you have 32 attention heads, you project 32 Query heads, 32 Key heads, and 32 Value heads.
In modern models (LLaMA 3, Mistral), we use Grouped-Query Attention (GQA). LLaMA 3 8B uses 32 Query heads but only 8 Key-Value head pairs. Every group of 4 Query heads shares a single Key/Value head. This reduces the KV cache memory footprint by 75%.
# Dimensions for LLaMA-3-8B
# Query: 32 heads * 128 dim = 4096
# Key/Value: 8 heads * 128 dim = 1024
q = x_norm @ W_q # Shape: [1, 32, 128]
k = x_norm @ W_k # Shape: [1, 8, 128]
v = x_norm @ W_v # Shape: [1, 8, 128]
# Apply RoPE to q and k for current position m
q, k = apply_rope(q, k, position=current_pos)
Why the KV Cache is Mandatory
Without a cache, to compute attention for the new token, you would have to re-project $K$ and $V$ for all previous 2,000 prompt tokens from scratch.
Instead, the model maintains a KV Cache in GPU memory. During this step:
- The newly computed $k$ and $v$ vectors (shape $[1, 8, 128]$) are appended to the layer's KV Cache buffer in VRAM.
- The layer now has complete keys $K_{\text{past}}$ and values $V_{\text{past}}$ for the entire sequence length $S$ (shape $[S, 8, 128]$).
Attention Math for a Single Token
The single query vector $q$ takes the dot product against all cached keys across the sequence:
$$\text{Scores} = \frac{q \cdot K_{\text{past}}^T}{\sqrt{d_k}} \quad (\text{Shape: } [32 \text{ heads}, 1, S])$$
$$\text{Weights} = \text{softmax}(\text{Scores}) \quad (\text{Probabilities over all } S \text{ past positions})$$
$$\text{Context} = \text{Weights} \cdot V_{\text{past}} \quad (\text{Shape: } [1, 32 \times 128] = [1, 4096])$$
The context vector is multiplied by an output projection matrix $W_o$ and added back to our original input via a residual skip connection:
$$x = x + (\text{Context} \cdot W_o)$$
C. SwiGLU Feed-Forward Network (MLP)
Roughly two-thirds of all model parameters reside in the Feed-Forward layers.
Modern models replace standard ReLU/GELU activations with SwiGLU (Swish-Gated Linear Units). It uses three separate weight matrices: $W_{\text{gate}}$, $W_{\text{up}}$, and $W_{\text{down}}$ with an intermediate hidden dimension of 14,336:
x_norm2 = rms_norm(x)
# Gate and Up projections
gate = silu(x_norm2 @ W_gate) # Shape: [1, 14336]
up = x_norm2 @ W_up # Shape: [1, 14336]
# Element-wise product and Down projection
hidden = gate * up # SwiGLU activation
out = hidden @ W_down # Shape: [1, 4096]
# Residual addition
x = x + out
This complete cycle (RMSNorm $\to$ GQA Attention $\to$ Residual $\to$ RMSNorm $\to$ SwiGLU $\to$ Residual) executes 32 times sequentially.
Step 3: Unembedding (The LM Head)
After exiting Layer 31, our vector $x$ has been updated 32 times and contains the contextual semantic representation of what should follow.
- Final RMSNorm: $x_{\text{final}} = \text{RMSNorm}(x) \in \mathbb{R}^{1 \times 4096}$.
- Projection to Vocabulary (LM Head): The vector multiplies against the unembedding weight matrix $W_{\text{vocab}} \in \mathbb{R}^{128256 \times 4096}$:
$$\text{Logits} = x_{\text{final}} \cdot W_{\text{vocab}}^T \in \mathbb{R}^{1 \times 128256}$$
The output is an array of 128,256 raw, unnormalized floating-point numbers (logits). Each number represents the model's uncalibrated score for every token in its vocabulary.
Logits preview for next token:
- Token "dog" (ID: 5642): +18.42
- Token "cat" (ID: 8921): +15.10
- Token "fox" (ID: 21094): +12.30
- Token "table" (ID: 7712): -4.80
... (128,252 other tokens)
Step 4: The Sampling Pipeline
How do we pick one token out of 128,256 candidates? Passing raw logits directly to an argmax would result in rigid, repetitive, deterministic output. Instead, logits pass through a sampling pipeline.
Raw Logits [128,256 floats]
│
▼
┌─────────────────────────────────┐
│ 1. Temperature Division (z / T) │
├─────────────────────────────────┤
│ 2. Top-K Truncation │
├─────────────────────────────────┤
│ 3. Top-P (Nucleus) Filtering │
├─────────────────────────────────┤
│ 4. Softmax Normalization │
├─────────────────────────────────┤
│ 5. Categorical Random Draw │
└─────────────────────────────────┘
│
▼
Selected Token ID: 5642 ("dog")
1. Temperature Scaling
Dividing logits by temperature $T$:
$$z_i = \frac{\text{logit}_i}{T}$$
- When $T < 1.0$ (e.g. 0.2): Differences between logits are amplified. High-probability tokens dominate, making output focused and deterministic.
- When $T > 1.0$ (e.g. 1.5): Differences are flattened. Less probable tokens gain higher relative probability, increasing randomness.
2. Top-K Filtering
Keep only the $K$ tokens with the highest logits (e.g., $K=50$). The logits for all remaining 128,206 tokens are set to $-\infty$.
3. Top-P (Nucleus) Filtering
Rather than a fixed number of tokens, Top-P selects the smallest set of tokens whose cumulative probability exceeds $P$ (e.g., $P=0.90$). If the top token has 92% probability, all other tokens are discarded immediately. If the top 20 tokens each have 4% probability, all 20 are retained.
4. Softmax Normalization
The surviving filtered logits are converted into true probabilities:
$$P(\text{token}_i) = \frac{e^{z_i}}{\sum_j e^{z_j}}$$
5. Categorical Sampling
A pseudo-random number generator draws a sample according to the normalized probability distribution.
If token 5642 ("dog") is drawn:
- The string
"dog"is streamed to your client interface. - The integer
5642is appended to the context sequence. - The next loop begins immediately with
token_id = 5642as input.
The Hardware Reality: Why Your $1,600 GPU Is 98% Idle
Here is the most critical systems engineering insight regarding LLM inference: during single-user token generation, you are almost never bottlenecked by GPU compute.
Let us run the math for an 8-billion parameter model in 16-bit precision (FP16):
- Model Weight Size: $8 \times 10^9 \text{ parameters} \times 2 \text{ bytes} = 16 \text{ GB}$.
- FLOPs per Token: Each parameter undergoes 1 multiply and 1 add per forward pass = $2 \times 8 \times 10^9 = 16 \text{ GFLOPs} = 0.016 \text{ TFLOPs}$.
-
Hardware (NVIDIA RTX 4090):
- Compute Capacity: 82.6 TFLOPS (FP16/FP32 Tensor Cores).
- Memory Bandwidth: 1,008 GB/s (GDDR6X).
To generate a single token, the GPU memory controller must stream the entire 16 GB of weights from VRAM across the memory bus into the processor registers:
$$\text{Minimum Time to Read Weights} = \frac{16 \text{ GB}}{1,008 \text{ GB/s}} \approx 0.01587 \text{ seconds } (15.87 \text{ ms})$$
$$\text{Theoretical Max Generation Speed} = \frac{1}{0.01587 \text{ s}} \approx 63 \text{ tokens/second}$$
Now calculate how much compute the GPU actually used during those 15.87 milliseconds:
$$\text{Actual Compute Rate} = \frac{16 \text{ GFLOPs}}{0.01587 \text{ s}} \approx 1.008 \text{ TFLOPS}$$
$$\text{Compute Utilization} = \frac{1.008 \text{ TFLOPS}}{82.6 \text{ TFLOPS}} \approx 1.22\%$$
Your GPU Tensor Cores are sitting completely idle 98.7% of the time, waiting for weights to travel over the memory bus.
RTX 4090 Single-Token Generation Breakdown:
┌─────────────────────────────────────────────────────────────┐
│ [██] Active Math (1.2%) │
│ [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] │
│ Waiting on VRAM Memory Bus Transfer (98.8%) │
└─────────────────────────────────────────────────────────────┘
How Modern Inference Engines Beat the Memory Wall
Understanding this memory-bandwidth bottleneck explains every major optimization in LLM serving infrastructure:
1. Weight Quantization (4-Bit / AWQ / GGUF)
If you quantize the 8B model to 4-bit weights:
- Weight size drops from 16 GB to ~4.5 GB.
- Memory transfer time drops from 15.8 ms to 4.4 ms.
- Token generation speed jumps from ~63 tokens/sec to ~225 tokens/sec on the exact same GPU.
2. Continuous Batching (vLLM / TGI)
If you batch 32 user requests together ($B=32$):
- You still load the 16 GB model weights from VRAM once per step.
- But you multiply those weights against 32 tokens simultaneously instead of 1.
- Arithmetic intensity increases 32x: the GPU does 512 GFLOPs per memory sweep, boosting compute utilization from 1.2% to nearly 40% with minimal increase in latency.
3. Speculative Decoding
A tiny, ultra-fast draft model (e.g. 1B parameters) guesses the next 4 tokens sequentially at high speed.
The large 8B target model then evaluates all 4 guessed tokens in a single parallel forward pass (which is compute-efficient). If the guesses are correct, you generate 4 tokens in the memory-time cost of 1.
Summary Mental Model
When you watch an LLM generate text:
- It does not think in words: It translates an integer token ID into a 4096-dimensional geometric coordinate.
- It preserves history in VRAM: It queries past conversation context through a pre-computed KV cache across 32 transformer layers.
- It outputs probabilities, not certainties: The final layer produces 128,000 raw logits that are filtered, temperature-scaled, and sampled.
- It is constrained by memory speed, not compute: Every single token requires a full sweep of every byte of model weights across the GPU memory bus.
Top comments (0)