How Does an LLM Request and Response Cycle Work? A Full Walkthrough

In this blog post, we will see how an LLM request and response cycle works, from the second you hit send to the moment the last word lands on your screen. I will not throw a wall of transformer math at you. Instead, we will follow one single prompt through every stage of the journey, so nothing feels abstract.

A single prompt sent to a large language model passes through multiple distinct stages before a response appears on screen. The process begins with the client app building a structured JSON request, which then clears an API gateway before the input text is broken into numeric tokens and assembled into the model's context window.

The model then runs a forward pass through transformer layers to predict one token at a time, repeating this loop until an end-of-sequence signal is reached. Tokens stream back to the client as they are produced and are converted back into readable text incrementally, which explains the word-by-word appearance of responses.

I got curious about mapping this out properly while building iamspeed.dev, my browser based LLM benchmarking tool. Before I could measure things like tokens per second or time to first token, I had to understand exactly what happens between "you press Enter" and "words start appearing." Turns out that gap has a lot more steps than most people assume.

The example we will follow

Let's keep one example running through the whole post. Say you type this into a chat app:

You: What's the capital of India?

And a few seconds later, the model replies:

Assistant: The capital of India is New Delhi.

Simple enough on the surface. But that one exchange touches a client app, an API layer, a tokenizer, a neural network with billions of parameters, a sampling algorithm, and a streaming pipeline, all before those six words show up in your chat window. Let's go step by step.

Step 1: You hit send

The moment you press Enter, your chat app does not just fire off the raw sentence. It builds a structured request, usually JSON, that looks something like this:

{
  "model": "some-llm-model",
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": "What's the capital of India?" }
  ],
  "temperature": 0.7,
  "max_tokens": 100,
  "stream": true
}

Notice a few things here. The system prompt is bundled in even though you never typed it. Any earlier messages in the conversation get bundled in too, since the model has no memory of its own between calls. And stream: true is already set, which matters a lot later. This payload gets sent over HTTPS to an API endpoint.

Step 2: The API gateway

Your request does not go straight to a model. It first hits an API gateway that handles the boring but essential stuff:

• Validating your API key or session token
• Checking rate limits, so nobody floods the system
• Routing the request to the right model version and the right compute cluster, often based on region for lower latency
• Logging the request for billing and abuse monitoring

Think of this layer as the bouncer and the traffic cop combined. Only after it clears the request does anything resembling "AI" actually happen.

Step 3: Tokenization

Here is where things get interesting. The model does not read English. It reads tokens, which are chunks of text mapped to numbers. A tokenizer breaks your sentence apart, usually using a scheme like byte pair encoding.

Our example sentence, "What's the capital of India?", might get tokenized into something like:

["What", "'s", " the", " capital", " of", " India", "?"]

Each of these chunks maps to a specific integer ID from the model's vocabulary, something like [1780, 434, 262, 3139, 286, 4881, 30]. The exact split depends on the tokenizer the model uses, but the idea is always the same: text becomes numbers, because that is the only thing a neural network can actually compute on.

This step matters more than people realize. It is also why providers bill you per token instead of per word or per character. A short word can be one token, a rare word can be split into three or four.

Step 4: Building the context window

Your new tokens do not go in alone. The system stitches together:

• The system prompt tokens
• Any prior conversation history tokens
• Your new message tokens

All of this gets packed into the model's context window, which is just a fancy term for "how many tokens the model can look at in one shot." Each token also gets a positional encoding attached, so the model knows word order, since without it, "India of capital the" and "capital of India" would look identical to the network.

Step 5: The forward pass

Now the actual model runs. Your token sequence flows through dozens of transformer layers stacked on top of each other. Inside each layer, two things happen repeatedly:

1. Self-attention, where every token looks at every other token in the sequence and decides how much to "pay attention" to it. This is how the model knows that "capital" relates to "India" and not to some other country mentioned three messages ago.
2. Feed-forward processing, where each token's representation gets transformed further, layer after layer, refining what the model "understands" about the sequence so far.

After all layers, the model produces a probability distribution over its entire vocabulary, essentially a ranked guess of what the very next token should be. For our example, after processing "What's the capital of India?", the model's internal state is now primed to predict something like "The" as a strong first candidate.

Step 6: Sampling the next token

Here is the part that surprises people the first time they learn it: the model does not generate the whole sentence at once. It generates one token, feeds that token back into itself as part of the input, and generates the next one. This repeats until it decides to stop.

Parameters like temperature, top_p, and top_k control how the next token gets picked from that probability distribution. Lower temperature means the model plays it safe and picks the highest probability token almost every time. Higher temperature adds randomness, useful for creative writing, less useful for factual answers.

For our example, the autoregressive loop looks roughly like this:

Input so far: "What's the capital of India?"
→ predict: "The"

Input so far: "...India?" + "The"
→ predict: " capital"

Input so far: "...The" + " capital"
→ predict: " of"

... and so on, until:
"The capital of India is New Delhi." + <end-of-sequence token>

That end-of-sequence token is the model's way of saying "I am done," which is also how it knows when to stop rather than rambling forever.

Step 7: Streaming the response back

Remember stream: true from Step 1? This is where it pays off. Instead of waiting for the entire response to finish generating and sending it as one big blob, the server pushes each token to your client the moment it is produced, usually over Server-Sent Events or a chunked HTTP connection.

That is exactly why you see words appear on screen one at a time instead of the whole answer popping in at once. It is not a visual effect, it is literally showing you tokens as the model produces them.

Step 8: Detokenization and rendering

On the client side, each incoming token ID gets converted back into readable text, the reverse of Step 3. The chat app appends each piece to the message bubble as it arrives, re-renders the UI, and by the time the end-of-sequence signal shows up, you are looking at the complete sentence:

The capital of India is New Delhi.

Six words. Eight steps. All of it usually finishing in under two seconds.

Where latency actually lives

Given my performance engineering background, I cannot write this post without pointing at the clock. Two numbers matter a lot more than "the response took 3 seconds":

• Time to first token (TTFT): how long you wait between hitting send and seeing the very first word appear. This is dominated by Steps 1 through 5, especially the forward pass on a long context window.
• Inter-token latency: how quickly tokens keep arriving after the first one, which shapes how "smooth" the typing effect feels.

A model can have a slow TTFT but fast inter-token speed, or the reverse, and the two create very different user experiences even if the total time is identical. This is the exact gap I was trying to measure when building iamspeed.dev, and honestly, understanding this full life cycle is what made the benchmarking numbers actually make sense instead of just being numbers on a dashboard.

Check iamspeed.dev to measure the LLM performance.

Wrap up

So the next time you type a question into a chat app and watch the answer type itself out, you now know the full trip that sentence takes: request formation, gateway checks, tokenization, context assembly, a forward pass through a massive network, token-by-token sampling, streaming, and finally detokenization back into words you can read.

It looks like magic from the outside. From the inside, it is a very well engineered pipeline.

Happy Testing!

Have you ever watched an LLM response stream in and wondered what was happening under the hood? Let me know in the comments.