"How Does LLM Actually Work? From Prompt to Prediction"

Ravindranath Guptha K — Sat, 15 Aug 2026 06:19:13 +0000

Large Language Models have quickly become part of everyday software development.

We ask them to explain code, debug errors, generate tests, write Python scripts, summarize documentation, or help us understand an unfamiliar codebase.

Within seconds, we get a response that can feel surprisingly natural.

But what actually happens during those few seconds?

Suppose you type:

What is a build system?

The model doesn't simply search through a database for a stored answer, and it doesn't generate the entire response in one shot.

At the heart of an autoregressive LLM is a deceptively simple task:

Given the tokens I've seen so far, what token should come next?

Getting to that prediction, however, involves several layers of computation.

At a high level:

Prompt
   ↓
Tokens
   ↓
Embeddings
   ↓
Transformer
   ↓
Logits
   ↓
Next Token
   ↓
Repeat

Let's follow that journey.

1. Everything Starts With the Prompt

Consider:

What is a build system?

Humans immediately recognize the words and their meaning.

A neural network needs numbers.

Before the model can process the question, the text passes through a tokenizer.

2. Tokenization: Breaking Text Into Pieces

A tokenizer divides text into smaller units called tokens.

Conceptually, our prompt might become:

["What", " is", " a", " build", " system", "?"]

This is only an illustration. Actual tokenization depends on the tokenizer used by the model.

A token isn't necessarily a complete word. It might represent:

a complete word
part of a word
punctuation
whitespace combined with text
a number
part of an identifier
a programming-language symbol

Each token is mapped to an integer called a token ID.

Conceptually:

["What", " is", " a", " build", " system", "?"]

                 ↓

[3923, 374, 264, 1975, 1887, 30]

The IDs above are illustrative.

The important part is the transformation:

Human-readable text has become a sequence of numbers the model can process.

But token IDs themselves don't capture useful semantic relationships.

The number 1975, for example, doesn't inherently explain what build means.

That's where embeddings enter the picture.

3. Embeddings: Turning Tokens Into Vectors

Each token is mapped to an embedding—a high-dimensional vector of numbers.

Conceptually:

build → [0.12, -0.47, 0.83, 0.21, ...]

These useful representations are learned during model training rather than manually assigned by engineers.

Embeddings give the neural network much richer representations to work with.

But language depends heavily on context.

Consider:

The developer restarted the build because it failed.

What does it refer to?

A human reader can connect it with the build.

The model needs a mechanism for modeling relationships between tokens as well.

That's where self-attention becomes important.

4. Self-Attention: Which Other Tokens Matter?

Self-attention is one of the core ideas behind the Transformer architecture.

Instead of treating every token independently, attention allows each token's representation to incorporate information from other relevant tokens in the context.

A useful mental model is that each token is asking:

Which other tokens should influence my representation right now?

Under the hood, tokens are projected into three important vectors:

Query (Q)
Key (K)
Value (V)

An intuitive way to think about them is:

Query: What information am I looking for?

Key: What information might I match with?

Value: What information should I contribute if I'm relevant?

The standard scaled dot-product attention operation is:

Attention(Q, K, V) = softmax(QKᵀ / √dₖ)V

We'll explore the mathematics more deeply in a later article.

For now, the key idea is:

Attention allows the representation of a token to depend on its surrounding context.

Transformers also use multiple attention heads, allowing different kinds of relationships to be modeled in parallel.

5. Through the Transformer

Self-attention is part of the larger Transformer architecture.

Token representations pass through a stack of Transformer layers.

A simplified view:

Token Embeddings
       ↓
Self-Attention
       ↓
Feed-Forward Network
       ↓
Next Transformer Layer
       ↓
Self-Attention
       ↓
Feed-Forward Network
       ↓
      ...

Real Transformer blocks also contain mechanisms such as residual connections and normalization.

As the representations move through these layers, they become increasingly contextualized.

The representation associated with build, for example, is no longer simply a static representation of that token.

It now incorporates information about how build is being used in this particular context.

Eventually, the model reaches the central question:

What token should come next?

6. From Transformer Output to Logits

After processing the context, the model produces scores for possible next tokens.

These scores are called logits.

Imagine a tiny example:

Possible Token	Logit
A	7.2
The	5.8
Build	3.4
Software	2.7
Banana	-4.1

A real model may have a vocabulary containing tens or hundreds of thousands of tokens.

That means the model produces a large vector of scores at every generation step.

Higher logits generally indicate more plausible next tokens.

These scores can then be converted into a probability distribution using softmax.

Conceptually:

Token	Probability
A	55%
The	25%
Build	10%
Software	7%
Everything else	3%

Again, these numbers are illustrative.

Now the model has possible continuations and their relative likelihoods.

7. Choosing the Next Token

How the next token is chosen depends on the decoding strategy.

Generation can be influenced by techniques such as:

temperature
top-k sampling
top-p sampling
greedy or other decoding strategies

Suppose the model selects:

A

The sequence now becomes:

What is a build system? A

The model isn't finished.

It predicts again.

8. The Answer Grows One Token at a Time

The model now has:

What is a build system? A

It predicts another token.

Perhaps:

build

Now:

What is a build system? A build

Then:

system

Then:

is

Then:

a

The process continues until a stopping condition is reached.

Eventually, we might get:

A build system is software that automates the process of compiling, linking, testing, and packaging software.

This is autoregressive generation.

At each step:

Current Context
      ↓
Transformer
      ↓
Next-Token Distribution
      ↓
Select Token
      ↓
Append Token
      ↓
Repeat

This is one of the most important ideas for understanding LLMs:

The answer isn't normally generated all at once. It grows token by token.

9. What Happens When an LLM Generates Code?

The same basic process applies when an LLM generates software.

Suppose we ask:

Write a C++ function that returns the larger of two integers.

The model might produce:

int maxValue(int a, int b) {
    return a > b ? a : b;
}

From our perspective, it looks as though the model understood the requirement and produced a complete C++ function.

But the fundamental generation mechanism hasn't changed.

The output is still produced as a sequence of tokens.

Conceptually:

int
 → maxValue
 → (
 → int
 → a
 → ,
 → int
 → b
 → )
 → {
 → return
 → a
 → >
 → b
 → ?
 → a
 → :
 → b
 → ;
 → }

The actual token boundaries depend on the tokenizer.

What's interesting is how the existing context changes what becomes likely next.

After:

int maxValue(

tokens associated with a C++ function signature become plausible.

Later, after:

return a > b

the prompt, previously generated code, and patterns learned during training make certain continuations more plausible than unrelated ones.

The model doesn't suddenly switch to a dedicated C++ code-generation engine.

The same fundamental loop continues:

Prompt + Generated Code
          ↓
      Transformer
          ↓
Next-Token Distribution
          ↓
     Select Token
          ↓
     Append Token
          ↓
        Repeat

The same principle applies to Python, Rust, Java, SQL, shell scripts, and other programming languages.

Of course, generating a small function is much easier than generating or modifying a large software component.

For larger tasks, the model may need to maintain relationships involving:

function signatures
types
variable names
APIs
control flow
previously generated code
programming-language syntax
user requirements

across a much larger context.

That raises an interesting question:

If an LLM generates code token by token, how can it produce syntactically and sometimes logically consistent programs?

That's a topic worth exploring separately.

For now, the important takeaway is:

Natural language and source code look very different to us, but to an LLM they ultimately become sequences of tokens to be modeled and predicted.

10. Doesn't Repeating the Transformer Get Expensive?

Yes.

Suppose the model has already processed a long prompt and generated hundreds of tokens.

Recalculating all of the attention information for every previous token from scratch for each new token would involve substantial redundant computation.

One important optimization is the KV cache.

During self-attention, the model computes Key and Value representations for tokens it has already processed.

During autoregressive decoding, those representations can be cached and reused.

Conceptually:

Without KV Cache

Recompute previous K/V
        ↓
Process next token
        ↓
Generate
        ↓
Repeat

With caching:

With KV Cache

Reuse previous K/V
        ↓
Compute K/V for new token
        ↓
Generate
        ↓
Repeat

This reduces redundant computation during generation, although the growing cache also consumes memory as the context becomes longer.

KV caching deserves its own deeper discussion later.

11. From Prompt to Prediction

We can now put the complete journey together:

User Prompt
     ↓
Tokenization
     ↓
Token IDs
     ↓
Embeddings
     ↓
Transformer Layers
     ↓
Self-Attention
     ↓
Contextual Representations
     ↓
Logits
     ↓
Next-Token Distribution
     ↓
Token Selection
     ↓
Append Token
     ↓
Repeat
     ↓
Final Response

From the user's perspective, the interaction looks incredibly simple:

Ask a question → Get an answer

Behind that interaction is a repeated sequence of numerical transformations and next-token predictions.

12. Is an LLM Really "Just a Next-Token Predictor"?

You've probably heard this description:

LLMs are just next-token predictors.

There is an important truth behind it.

Next-token prediction is central to the training objective of many autoregressive LLMs, and autoregressive generation produces output by repeatedly predicting subsequent tokens.

But the word just hides most of what makes these systems interesting.

Predicting what follows:

The sky is...

may seem straightforward.

Now consider asking the same model to:

explain an algorithm
generate a C++ implementation
translate Python into Rust
summarize a technical document
analyze an error message
reason about unfamiliar code

The output still emerges token by token.

But producing useful next-token distributions requires a sophisticated learned representation of the context.

The simple question is:

What should come next?

The complicated part is the enormous learned system answering it.

Transformer layers, attention mechanisms, embeddings, billions of learned parameters, and highly optimized inference infrastructure all contribute to those predictions.

Then the model does it again.

And again.

Final Thoughts

An LLM can seem mysterious when all we see is:

Prompt → Answer

But once we look inside, the process becomes easier to reason about.

Text becomes tokens.

Tokens become numerical representations.

Transformers use attention to build contextual representations.

Those representations produce scores for possible next tokens.

A token is selected.

Then the process repeats.

That basic loop is the foundation for much of what we experience when interacting with modern language models.

Once these pieces become familiar, an LLM starts looking less like a mysterious black box and more like something engineers are used to understanding:

A very large system built from smaller, understandable components.

What's Next?

We've followed an LLM from the moment a prompt enters the system to the point where a response emerges, one token at a time.

But we moved quickly through the very first step: tokenization.

How does an LLM decide where one token ends and another begins? Why can a single word become multiple tokens? And what happens when the input is source code rather than ordinary English?

In the next article in this LLM Internals series, we'll take a closer look at:

Tokenization: How LLMs Break Text and Source Code Into Tokens

Once we understand that first transformation, we'll be ready to move deeper into what happens inside the model.

DEV Community: Ravindranath Guptha K

"How Does LLM Actually Work? From Prompt to Prediction"

1. Everything Starts With the Prompt

2. Tokenization: Breaking Text Into Pieces

3. Embeddings: Turning Tokens Into Vectors

4. Self-Attention: Which Other Tokens Matter?

5. Through the Transformer

6. From Transformer Output to Logits

7. Choosing the Next Token

8. The Answer Grows One Token at a Time

9. What Happens When an LLM Generates Code?

10. Doesn't Repeating the Transformer Get Expensive?

11. From Prompt to Prediction

12. Is an LLM Really "Just a Next-Token Predictor"?

Final Thoughts

What's Next?

Tokenization: How LLMs Break Text and Source Code Into Tokens