Tokens, Context, Parameters: The Real Mechanics of an LLM

Tokens, Context, and Parameters: How Large Language Models Actually Work

Large language models can answer questions, generate software, summarize documents, and produce human-like writing. These abilities emerge from three closely connected elements:

• Tokens represent the information entering and leaving the model.
• Context contains the information available during the current prediction.
• Parameters encode the statistical patterns learned during training.

Together, these components allow an LLM to predict one token at a time.

---

1. Tokens: The Model’s Units of Language

A language model does not process text directly as sentences or complete words. It first uses a tokenizer to divide text into smaller units called tokens.

A token can represent:

• A complete word
• Part of a word
• Punctuation
• A number
• Whitespace
• A special control symbol

For example:

The model is learning.

might become:

The · model · is · learn · ing · .

Breaking uncommon words into smaller pieces allows the model to represent a very large vocabulary without storing every possible word.

From text to token IDs

Every token is assigned an integer called a token ID:

"The"     → 464
" model"  → 2746
" is"     → 318
" learn"  → 2193
"ing"     → 278
"."        → 13

The exact IDs depend on the tokenizer used by the model.

Token IDs are only labels. Before they can be processed mathematically, the model converts them into vectors.

Token embeddings

The model contains a learned embedding matrix:

Vocabulary size × embedding dimension

If the vocabulary contains 50,000 tokens and the model’s internal dimension is 4,096, the embedding matrix has the shape:

50,000 × 4,096

Each row is a learned numerical representation of one token. Looking up a token ID selects the corresponding row from the matrix.

A token therefore becomes a vector such as:

[0.14, -0.82, 0.37, ..., 0.09]

The individual values usually do not have simple human-readable meanings. Meaning is distributed across many dimensions.

Position information

Transformers process multiple tokens in parallel, so they also need information about token order. The model adds or applies a positional representation to each token embedding.

Conceptually:

Input representation = token embedding + position information

This helps the model distinguish between sentences such as:

The dog chased the cat.

and:

The cat chased the dog.

They contain similar tokens, but their order changes their meaning.

---

2. Context: The Model’s Temporary Working Information

The context is the complete sequence of tokens available to the model during a prediction.

It can include:

• System instructions
• The user’s prompt
• Previous conversation messages
• Retrieved information
• Uploaded document content
• Tool results
• Tokens already generated in the response

A model’s context window defines the maximum number of tokens it can process at one time.

Conceptually:

Instructions + conversation + documents + generated output ≤ context window

If the combined content exceeds the limit, some information must be removed, compressed, summarized, or handled through another retrieval mechanism.

Context is not permanent memory

Context acts more like temporary working memory than permanent storage. Information inside the active window can influence the next prediction. Information outside it normally cannot unless it is supplied again.

This creates an important distinction:

Parameters = patterns learned during training
Context    = information supplied for the current task

The context changes from one conversation to another. The model’s parameters normally remain fixed during ordinary inference.

Causal attention

Most text-generating LLMs use causal attention. When predicting a token, the model may attend to previous tokens but not future tokens.

For a sequence:

The capital of Japan is

the representation at the final position can use information from all preceding positions. It cannot use the answer token because that token has not been generated yet.

A causal mask enforces this direction:

Token 1 can attend to: 1
Token 2 can attend to: 1, 2
Token 3 can attend to: 1, 2, 3
Token 4 can attend to: 1, 2, 3, 4

This is what makes autoregressive generation possible.

---

3. Parameters: The Model’s Learned Numerical Structure

Parameters are the trainable numerical values inside the neural network. They include large matrices and smaller vectors used throughout the transformer.

The symbol θ, pronounced “theta,” is often used to represent all model parameters collectively.

A model with billions of parameters contains billions of learned floating-point values. These parameters do not form a searchable database of sentences. Instead, information is distributed across many interacting weights.

Where the parameters are located

A transformer language model typically contains parameters in several major components.

Embedding matrix

The embedding matrix converts token IDs into vectors:

token ID → embedding vector

These embeddings are learned during training.

Attention projections

Inside each self-attention layer, the input representation is transformed into queries, keys, and values:

Q = XWQ
K = XWK
V = XWV

Where:

• X is the current sequence representation.
• WQ is the query projection matrix.
• WK is the key projection matrix.
• WV is the value projection matrix.

Attention scores are calculated using:

Attention(Q, K, V) = softmax(QKᵀ / √dk)V

The result is commonly passed through another learned matrix, WO.

These matrices let the model determine which previous positions are relevant to the current token.

Multi-head attention

Transformers usually divide attention into multiple attention heads. Each head has separate projection weights and can learn different relationships.

Different heads may become sensitive to patterns involving:

• Nearby words
• Long-distance dependencies
• Grammatical structure
• References between names and pronouns
• Formatting or positional relationships

The heads are combined and projected back into the model’s main representation space.

Feed-forward network

Each transformer block also contains a feed-forward network, often called an MLP.

A simplified version is:

MLP(X) = activation(XW1 + b1)W2 + b2

The first projection usually expands the representation into a larger intermediate dimension. The second projects it back to the model dimension.

Modern models may use gated structures such as SwiGLU, but the purpose is similar: transform the representation at each token position using learned nonlinear features.

Layer normalization

Layer-normalization components help keep internal activations numerically stable. They may contain learned scaling and shifting parameters.

Output projection

After the final transformer layer, the model converts the hidden representation into one score, or logit, for every possible token:

hidden state → vocabulary logits

A softmax function converts these logits into a probability distribution:

P(next token | previous tokens; θ)

The model can then select or sample the next token from this distribution.

---

4. Inside a Transformer Block

An LLM contains many transformer blocks stacked on top of one another. Each block gradually refines the representation of every token.

A simplified block contains:

1. Layer normalization
2. Multi-head self-attention
3. A residual connection
4. Another normalization stage
5. A feed-forward network
6. Another residual connection

The general flow is:

Input representations
        ↓
Layer normalization
        ↓
Self-attention
        ↓
Residual addition
        ↓
Layer normalization
        ↓
Feed-forward network
        ↓
Residual addition
        ↓
Output representations

Residual connections

Residual connections add a block’s input to its output:

output = input + transformation(input)

They help information and gradients travel through deep networks. Without them, training models with many layers would be considerably more difficult.

What changes through the layers?

Early layers may identify local or surface-level patterns. Deeper layers can combine information across positions and represent more abstract relationships.

The model’s internal representation of a token therefore depends on both:

• The token’s original embedding
• The surrounding tokens in the current context

The word “bank,” for example, may develop different contextual representations in:

She deposited money at the bank.

and:

They sat on the river bank.

---

5. How an LLM Is Trained

Before an LLM can generate useful text, its parameters must be learned from data.

The basic training objective is next-token prediction.

Step 1: Prepare the training data

Training text is cleaned, filtered, and converted into token sequences.

Suppose a training sequence is:

Large language models predict tokens

The model receives shifted input-and-target pairs:

Input:  Large
Target: language

Input:  Large language
Target: models

Input:  Large language models
Target: predict

In practice, many positions are trained simultaneously.

Step 2: Run a forward pass

The token sequence passes through the embedding layer and transformer blocks.

For every position, the model produces a probability distribution over the vocabulary.

For example:

"tokens"      0.61
"text"        0.14
"words"       0.09
"outputs"     0.04
other tokens  0.12

Step 3: Measure the error

The predicted distribution is compared with the actual next token using cross-entropy loss.

For the correct token y, the loss can be written as:

L = −log P(y | previous tokens; θ)

If the model gives the correct token a high probability, the loss is low. If it assigns the correct token a low probability, the loss is high.

Step 4: Backpropagate the error

The training system uses backpropagation to calculate how much each parameter contributed to the error.

These derivatives form the gradient:

∇θL

The gradient indicates which direction would increase the loss. Training moves the parameters in the opposite direction.

Step 5: Update the parameters

An optimizer updates the weights:

θ ← θ − η∇θL

Where:

• θ represents the model’s parameters.
• L represents the loss.
• ∇θL represents the loss gradient.
• η represents the learning rate.

Real training usually employs optimizers such as Adam or AdamW rather than basic gradient descent, but the underlying idea is the same.

Step 6: Repeat across many batches

This process is repeated across enormous numbers of token sequences:

Forward pass
    ↓
Calculate loss
    ↓
Backpropagation
    ↓
Parameter update
    ↓
Next batch

Over time, the model becomes better at predicting tokens across many types of text.

---

6. Pretraining, Fine-Tuning, and Alignment

LLM development commonly involves several training stages.

Pretraining

During pretraining, the model learns general language patterns from a large and diverse dataset through next-token prediction.

It may learn patterns involving:

• Grammar and syntax
• Semantic relationships
• Writing styles
• Factual associations
• Programming languages
• Common reasoning structures

Pretraining creates the model’s broad foundational capabilities.

Supervised fine-tuning

The pretrained model may then be trained on curated examples containing prompts and high-quality responses.

This teaches it to follow instructions and produce answers in more useful formats.

Preference optimization

Human or AI-generated preference data can be used to improve helpfulness, safety, and response quality.

Possible approaches include:

• Reinforcement learning from human feedback
• Direct Preference Optimization
• Related preference-learning techniques

These stages still modify parameters, but they use more targeted data and objectives than general pretraining.

---

7. How Generation Works After Training

During normal use, the model performs inference. Its learned parameters are usually fixed.

Suppose the prompt is:

The capital of Japan is

The generation process is:

1. The tokenizer converts the text into token IDs.
2. The embedding layer converts the IDs into vectors.
3. Position information is incorporated.
4. Transformer layers process the context.
5. The output layer produces vocabulary logits.
6. Softmax converts the logits into probabilities.
7. A decoding strategy chooses the next token.
8. The chosen token is added to the context.
9. The process repeats.

The model might produce:

Tokyo

After generating Tokyo, it predicts the token that should follow it.

Decoding strategies

The model does not always select the single highest-probability token. Generation can be controlled using methods such as:

• Greedy decoding: choose the most probable token.
• Temperature: adjust how sharp or varied the probability distribution is.
• Top-k sampling: sample only from the k most likely tokens.
• Top-p sampling: sample from the smallest group whose combined probability reaches a threshold.

These settings affect how the model selects output tokens. They do not retrain its parameters.

---

8. Parameters Versus Context

Parameters and context influence the same prediction in different ways.

Feature	Parameters	Context
Purpose	Store learned statistical patterns	Supply information for the current prediction
Origin	Learned during training	Provided at inference time
Duration	Persists across prompts	Temporary and task-specific
Typical scale	Millions or billions of values	Thousands or millions of tokens
Changed by normal prompting?	No	Yes
Example	Learned language structure	The current conversation

This distinction explains why a model can adapt to instructions without changing its weights. The instructions alter the context, which changes the model’s internal activations and predictions.

---

9. What Parameter Count Does—and Does Not—Mean

A parameter count measures the model’s numerical capacity. A larger model can potentially represent more complex patterns, but size alone does not determine performance.

Quality also depends on:

• Training-data quality and diversity
• Model architecture
• Tokenizer design
• Training duration
• Optimization methods
• Context handling
• Fine-tuning and alignment
• Inference techniques

A smaller model trained efficiently on appropriate data can outperform a larger model on a specialized task.

Parameter count should therefore be treated as one engineering characteristic—not a complete measure of intelligence or quality.

---

10. Common Misconceptions

“One token equals one word”

Not necessarily. A word may be one token, several tokens, or part of a larger token.

“The context window is permanent memory”

It is temporary information available during the current inference process.

“Parameters contain copies of documents”

Parameters encode distributed numerical patterns. They are not equivalent to files or database records, although models can sometimes reproduce memorized material.

“The model searches its training data for every answer”

During standard inference, the model calculates predictions using its parameters and current context. External search or retrieval occurs only when a separate tool or retrieval system is connected.

“The model learns from every conversation immediately”

Ordinary conversations usually do not update the model’s parameters. Updating parameters requires a separate training process.

“More parameters always produce a better model”

Model size is only one factor. Architecture, data, training, and task fit are equally important.

---

The Complete Picture

The complete LLM pipeline can be summarized as:

Text
  ↓
Tokenizer
  ↓
Token IDs
  ↓
Token embeddings + position information
  ↓
Transformer layers
  ├─ Self-attention
  ├─ Feed-forward networks
  ├─ Normalization
  └─ Residual connections
  ↓
Vocabulary logits
  ↓
Probability distribution
  ↓
Next token

During training, prediction errors are used to update the parameters:

Training data
  ↓
Next-token prediction
  ↓
Cross-entropy loss
  ↓
Backpropagation
  ↓
Optimizer update
  ↓
Improved parameters

During inference, those learned parameters are reused without normally being changed:

Current tokens + context + learned parameters
                         ↓
              next-token probability

The central idea is simple:

Tokens carry the information, context determines what the model can currently consider, and parameters define the learned transformations used to predict what comes next.

At Engineer Philosophy, our objective is not simply to connect an interface to an LLM. We design the complete information flow around the model—controlling the tokens it receives, constructing useful context, selecting the appropriate model, and validating its output for the intended application.