You Typed a Prompt. What Happens Next?

You open a chat window and type:

The mole on my face may need a biopsy. Explain this simply.

Then you press Send.

A few moments later, an answer begins appearing on your screen.

What happened between those two moments?

The model did not search a dictionary for the meaning of every word. It did not read the prompt exactly as a person would. It also did not write the complete answer internally and reveal it all at once.

Instead, your prompt went through a pipeline:

messages → tokens → embeddings → Transformer layers
         → logits → probabilities → one new token
         → repeat until finished

Let us follow this one prompt through that journey.

Before we begin: training has already happened

Everything in this article happens during inference—the stage where an already-trained model responds to an input.

The model’s weights, embedding matrix, attention projections, and output matrix were learned earlier during training. They normally remain fixed while your request is being answered.

Your prompt affects temporary calculations inside the model. It does not immediately retrain the model or permanently rewrite its weights.

With that out of the way, let us press Send.

Step 1: Your prompt joins the conversation

You see only the message you typed, but a chat application usually manages a list of messages with roles:

System: You are a helpful assistant.
User: The mole on my face may need a biopsy. Explain this simply.
Assistant:

If this is an ongoing conversation, earlier messages may also be included. An application may additionally provide retrieved documents, tool results, or other instructions.

The model does not intrinsically understand JavaScript-like objects containing role and content. Before inference, a chat template converts the messages into one sequence, using special control tokens to identify message boundaries and roles.

A simplified sequence might resemble:

<system>
You are a helpful assistant.
</system>
<user>
The mole on my face may need a biopsy. Explain this simply.
</user>
<assistant>

The exact control tokens differ between model families. Hugging Face’s chat-template documentation makes the central point clearly: even a chat model ultimately receives and continues a sequence of tokens.

The final <assistant> marker tells a chat-tuned model that an assistant response should come next.

Step 2: The tokenizer breaks the sequence into tokens

The model cannot process the raw string directly. A tokenizer converts it into smaller text units called tokens.

Tokens are not always complete words. Depending on the tokenizer, the sequence might be divided approximately like this:

The | mole | on | my | face | may | need | a | bio | psy | .
Explain | this | simply | .

That split is only illustrative. A real tokenizer may split the same text differently.

Tokens can represent:

• complete words;
• parts of words;
• punctuation;
• spaces or space-prefixed pieces;
• bytes or characters;
• special control markers.

Every token belongs to the tokenizer’s fixed vocabulary and has an integer ID:

"The"     → 791
"mole"    → 17342
"face"    → 3234
"biopsy"  → 48291

These numbers are illustrative too.

The vocabulary is not a dictionary. It does not store definitions such as:

mole → a mark on the skin

It stores a mapping closer to:

text piece → token ID

Subword methods such as BPE, WordPiece, and Unigram allow a limited vocabulary to represent a much larger range of text. The Hugging Face tokenizer guide compares these approaches.

At the end of this step, the application has a sequence of IDs:

[system-token, 1639, 389, ..., user-token, 791, 17342, ..., assistant-token]

The token IDs themselves do not contain useful semantic meaning. They are indexes telling the model which rows to retrieve next.

Step 3: Token IDs become embeddings

Inside the model is a learned table called the embedding matrix.

It contains one starting vector for every vocabulary token:

Token	Starting embedding
mole	[0.18, -0.42, 0.71, ...]
face	[-0.14, 0.63, 0.27, ...]
biopsy	[0.51, 0.09, -0.32, ...]

If the vocabulary contains V tokens and the model’s hidden size is d, the embedding matrix has the shape:

V × d

For example, a model could have 50,000 vocabulary tokens and vectors containing 4,096 values.

When the tokenizer produces the ID for mole, the model retrieves the corresponding row from this table.

token "mole"
      ↓
token ID
      ↓
row in embedding matrix
      ↓
starting vector

This is why it is called an embedding: a discrete token is placed into a continuous, high-dimensional numerical space.

But mole can have several meanings

The embedding matrix still contains only one starting vector for the token mole, even though the word can mean:

• an animal;
• a mark on the skin;
• a spy.

That stored vector is only the starting point. Context will be incorporated as the sequence passes through the Transformer.

The stored row itself is not rewritten for this prompt. The model creates new temporary hidden states during the forward pass.

Step 4: The model adds positional information

The sequences below contain the same words but do not mean the same thing:

dog bites man
man bites dog

The model therefore needs information about where each token occurs.

Different Transformer families represent position differently. The original Transformer added sinusoidal positional encodings, while later models may use learned, relative, or rotary position techniques.

The exact technique is less important for this mental model than its purpose:

Every token enters the Transformer with information about both its identity and its position.

Step 5: The prompt enters the prefill phase

Inference for a decoder-only LLM is often easier to understand as two phases:

1. Prefill: process the prompt that already exists.
2. Decode: generate new tokens one at a time.

During prefill, the model processes the prompt tokens through all its Transformer layers. Because all prompt tokens are already known, their calculations can be performed largely in parallel.

However, the model still uses a causal mask. A position may use only itself and earlier positions.

Position 1 can see: 1
Position 2 can see: 1, 2
Position 3 can see: 1, 2, 3
...
Final prompt position can see: the complete prompt before it

This creates an important correction to a common explanation.

Does the mole token look ahead at face and change its meaning?

Not in a decoder-only model if face appears after mole.

At the mole position, causal attention prevents the model from seeing future tokens such as face and biopsy. Its hidden state is not later rewritten by those future words.

Instead, later positions can see mole:

"mole" position   → can see "The mole"
"face" position   → can see "The mole on my face"
"biopsy" position → can see everything before and including "biopsy"
final position     → can see the complete prompt

The final prompt position therefore contains a representation built from the entire preceding prompt. That is the position used to predict the first response token.

In an encoder-only model such as BERT, attention can be bidirectional, so the mole position itself can attend to words on both sides. That is not how causal attention in a GPT-style decoder works.

Step 6: Attention lets positions retrieve relevant context

Every Transformer layer creates three projections from each current hidden state:

• Query: what information is this position looking for?
• Key: what kind of information does this position offer?
• Value: what information should this position contribute if selected?

The model compares a query with the keys of the visible positions. After scaling, masking, and softmax, those comparisons become attention weights. The output is a weighted combination of the value vectors.

The original Transformer expresses this as:

Attention(Q, K, V) = softmax(QKᵀ / √dk) V

This mechanism comes from Attention Is All You Need.

For the final prompt position, different attention heads may retrieve information connected to:

• mole, face, and biopsy for the topic;
• explain for the requested action;
• simply for the desired style;
• earlier system instructions for the expected behaviour.

Attention does not search through every token in the model’s vocabulary. It operates over the token representations in the available sequence.

It is also not literal, permanent memory. A better description is:

Attention is a context-dependent information-routing mechanism.

Step 7: The representations pass through many layers

A Transformer layer contains more than attention. A simplified decoder block includes:

hidden states
    ↓
normalization
    ↓
masked self-attention
    ↓
residual connection
    ↓
normalization
    ↓
feed-forward network / MLP
    ↓
residual connection

The precise ordering varies between model architectures.

Attention moves and combines information across token positions. The feed-forward network performs additional learned processing at each position. Residual connections preserve information across deep stacks, and normalization helps keep computation stable.

This block is repeated many times.

Therefore, a token position does not have only one vector throughout the model. It has:

starting embedding
      ↓
hidden state after layer 1
      ↓
hidden state after layer 2
      ↓
...
      ↓
final hidden state

These hidden states exist temporarily for this inference request. Hugging Face models can expose the initial embedding output and the hidden states produced by individual layers, as shown in the GPT-2 model documentation.

Step 8: The last prompt position becomes the prediction point

After prefill passes through the final Transformer layer, the model has a final hidden state for every prompt position.

The important one for generation is the final input position—often the last token of an assistant-generation marker added by the chat template.

Why that position?

Because in a causal model it can access everything before it:

system instructions
+ conversation history
+ the current user prompt
+ the fact that an assistant response should begin

Its final hidden vector is a compressed, context-dependent representation used to answer:

Given everything before this point, which token should come next?

The model does not need to turn this vector into a human-readable summary first. It sends it directly to the output layer.

Step 9: The unembedding matrix produces logits

At the beginning, the embedding matrix performed this operation:

token ID → hidden-sized vector

Now the model needs the opposite direction:

final hidden vector → one score for every vocabulary token

A learned output projection performs this operation:

logits = final hidden state × output matrix + bias

This output projection is often called the:

• language-model head;
• LM head;
• output projection;
• or informally, the unembedding matrix.

If the hidden size is d and vocabulary size is V, its conceptual shape is:

d × V

Some models reuse the transpose of the input embedding matrix here. This is called weight tying. Other models use separate parameters.

For our prompt, the output might contain illustrative scores like:

Candidate next token	Logit
It	9.1
A	8.4
This	7.9
Banana	-3.2

A logit is a raw, unnormalized score. It is not yet a probability.

The output layer produces a score for every vocabulary token. This is the first point in this journey where the complete output vocabulary is considered at once.

Step 10: Logits become a probability distribution

Before selecting a token, the inference system may adjust the logits. The precise implementation varies, but temperature and repetition penalties are commonly applied before or as part of building the sampling distribution.

Temperature

Temperature controls how sharp the probability distribution becomes:

• lower temperature makes high-scoring tokens more dominant;
• higher temperature gives lower-scoring alternatives more opportunity.

Temperature does not add knowledge or creativity to the model. It changes the randomness of token selection.

Presence and frequency penalties

Some APIs expose repetition controls:

• a presence penalty reacts to whether a token has appeared;
• a frequency penalty reacts to how often it has appeared.

These controls normally make repetition less likely. They do not guarantee that a previous token will never appear again, and they do not update the model’s training data.

Softmax transforms logits into a probability distribution that adds up to 1:

Candidate	Illustrative probability
It	52%
A	27%
This	17%
Other tokens	4%

Step 11: A decoding strategy chooses one token

The system now chooses a token according to its decoding strategy.

• Greedy decoding selects the highest-probability token.
• Sampling selects from a probability distribution instead of always taking the maximum.

Two common sampling controls limit the candidate set:

• Top-k keeps the k highest-probability candidates.
• Top-p keeps the smallest group of candidates whose cumulative probability reaches a threshold.

After filtering, the remaining probabilities are normalized for sampling. Exact processing order and formulas are implementation-specific. Hugging Face documents temperature, top-k, and top-p in its text-generation guide.

Suppose the selected token is:

It

The model has generated exactly one token—not the full answer.

Step 12: The selected token is appended

The sequence now looks conceptually like this:

<system> ... </system>
<user> The mole on my face may need a biopsy. Explain this simply. </user>
<assistant> It

Now the model must predict the token after It.

The generated token receives its embedding and positional information, passes through the Transformer layers, produces another final hidden state, and is projected into a new set of vocabulary logits.

Perhaps the next token is means:

It → means

Then the model runs again:

It → means → your → doctor → wants → ...

This is why decoder-only generation is autoregressive: every generated token becomes part of the input used to generate the following token.

Step 13: The KV cache prevents wasteful repetition

If the model recalculated every earlier prompt token from scratch after generating each new token, generation would be extremely wasteful.

Attention uses key and value vectors for previous positions. During prefill, the system stores those vectors in a KV cache.

During decoding:

Generate "It"
  → reuse cached K/V for the prompt
  → store K/V for "It"

Generate "means"
  → reuse cached K/V for prompt + "It"
  → store K/V for "means"

The model still calculates the new token’s query, key, value, and subsequent layer operations. It simply avoids recalculating the stored keys and values for the previous sequence.

The cache therefore trades memory for speed. It grows as more tokens are stored, although sliding-window and other attention designs may limit that growth.

A KV cache does not automatically give the model a larger context window. It makes generation within the supported context more efficient. Hugging Face’s KV-cache guide describes this reuse and the available speed-versus-memory strategies.

Step 14: Tokens are decoded and streamed to your screen

The model generates token IDs. The tokenizer’s decoding process converts those IDs back into readable text.

[2181, 3445, 701, ...]
        ↓
"It means your..."

Applications often stream decoded text as tokens or small groups of tokens become available. That is why you see the response appear gradually instead of waiting for the entire paragraph.

Tokenizer decoding should not be confused with a Transformer decoder. The names are similar, but they refer to different operations:

• tokenizer decoding: token IDs → text;
• Transformer decoder: causally processes token representations and generates subsequent tokens.

Step 15: The model eventually stops

The generation loop continues until a stopping condition occurs. Common conditions include:

• the model generates an end-of-sequence or end-of-message token;
• the system encounters a configured stop sequence;
• the maximum output-token limit is reached;
• the application stops generation for another reason.

The final generated sequence is decoded, and you see the completed response.

Where does the context window fit?

Everything sent to the model must fit within its supported context budget. Depending on the application, that can include:

system instructions
+ conversation history
+ retrieved documents
+ tool results
+ current prompt
+ generated response

The context window is not permanent memory. It is the amount of tokenized information available to the model for this inference sequence.

If information is not present in the current context—and is not recovered by an external retrieval or memory system—the model cannot directly attend to it.

The context window also should not be confused with the KV cache:

Concept	Purpose
Context window	Limits how much tokenized sequence the model can process
KV cache	Stores previous attention keys and values to speed up decoding

What did not happen?

Several tempting explanations are slightly wrong.

The model did not fetch definitions from a dictionary

The tokenizer vocabulary maps text pieces to IDs. Learned meaning is distributed across embeddings and the model’s other parameters.

The model did not permanently modify the mole embedding

It retrieved the stored starting embedding and created temporary hidden states while processing this request.

In a decoder-only model, mole did not look ahead at face

Causal attention blocks future positions. Later positions combined the earlier mole token with face, biopsy, and the rest of the prompt.

Attention did not compare the prompt against the complete vocabulary

Attention combined visible sequence representations. The LM head later produced logits for the full output vocabulary.

The model did not generate the complete answer in one operation

It generated one token, appended it, and repeated the process.

The KV cache did not increase the context window

It reused earlier key and value tensors so they did not need to be recomputed.

The model did not train itself on your message while answering

Ordinary inference uses fixed trained parameters. Your message influences temporary activations and the current output.

The entire journey in one view

1. You press Send.

2. The application assembles the conversation:
   system instructions + history + current prompt + assistant marker

3. A chat template converts the messages into one model-specific sequence.

4. The tokenizer converts text pieces into token IDs.

5. The embedding matrix converts every ID into a starting vector.

6. Positional information tells the model where each token occurs.

7. During prefill, causal Transformer layers process the prompt and build
   temporary hidden states and a KV cache.

8. The final prompt position contains information gathered from the complete
   preceding prompt.

9. The LM head / unembedding matrix converts that final hidden state into
   one logit for every vocabulary token.

10. Temperature or repetition controls may adjust the logits, and softmax
    converts them into probabilities.

11. A decoding strategy may filter the candidates and selects one token.

12. The new token is appended to the sequence.

13. During decoding, cached keys and values are reused while one new token is
    processed at a time.

14. The tokenizer converts generated token IDs back into text, which may be
    streamed to your screen.

15. Generation ends when a stopping condition is reached.

The shortest accurate explanation is:

When you send a prompt, an LLM converts it into token vectors, processes those vectors through causally masked Transformer layers, uses the final prompt position to score every possible next token, selects one, appends it, and repeats—reusing cached attention information—until the response is complete.

Sources and further reading

• Vaswani et al., Attention Is All You Need
• Hugging Face: Chat templates
• Hugging Face: Tokenization algorithms
• Hugging Face: GPT-2 hidden states and language-model logits
• Hugging Face: Text-generation controls
• Hugging Face: KV-cache strategies

---

Whenever the terminology becomes overwhelming, return to this smaller loop:

prompt → tokens → temporary representations → next-token probabilities
       → selected token → repeat

That loop is the heart of decoder-only LLM inference.