DEV Community

Shrijith Venkatramana
Shrijith Venkatramana

Posted on

A Developer's Guide to LLM Model Serving - Part 2

#ai #webdev #programming #productivity

Hello, I'm Shrijith Venkatramana. I'm building git-lrc, an AI code reviewer that runs on every commit. Star Us to help devs discover the project. Do give it a try and share your feedback for improving the product.

There are LLM inference enthusiasts who talk a lot about LLM serving.

People mention terms like prefill, decode, KV cache, continuous batching, and paged attention. After reading several articles, I could recognize the words, but I still could not answer a basic question:

What actually happens after I send a prompt to an LLM?

As software engineers, we usually learn systems as a sequence of events. A web request reaches a server, middleware runs, business logic executes, and a response returns.

I wanted the same mental model for LLMs.

This article covers the second part of that journey.

1. The Request Begins

A simplified request flow looks like this:

Client
    ↓
Load Balancer
    ↓
LLM Server
    ↓
Tokenizer
    ↓
Prefill
    ↓
Decode
    ↓
Streaming Response
Enter fullscreen mode Exit fullscreen mode

The two important stages are:

  • Prefill
  • Decode

Once these two ideas become clear, many later topics fit into place.

2. What Is Prefill?

Suppose the prompt is:

Explain quantum computing like I'm ten.

My first question was:

Does the model start generating the response immediately?

The answer is simpler than I expected.

The model first processes the entire prompt.

This stage is called prefill.

A useful mental model is:

kvCache, firstToken := Prefill(prompt)
Enter fullscreen mode Exit fullscreen mode

Prefill performs two tasks:

  • Build the initial KV cache.
  • Generate the first output token.

The model now has enough information to continue generating the response.

3. Decode Is a Simple Loop

After prefill, the rest of the response comes from a repeated operation.

cache, token := Prefill(prompt)

for token != EOS {
    emit(token)
    cache, token = Decode(cache, token)
}
Enter fullscreen mode Exit fullscreen mode

Each call to Decode() performs three actions:

  1. Generate the next token.
  2. Update the KV cache.
  3. Return the new cache.

That loop continues until the model produces an end-of-sequence token.

This was the first point where LLM inference started to feel like ordinary software instead of a black box.

4. What Is the KV Cache?

The name made me think of a cache like Redis or a Go map.

The actual idea is much simpler.

The KV cache is data stored in GPU memory.

During prefill, the model computes internal data for every prompt token and stores it.

During decode, the model adds one new entry for every generated token.

A simple mental model looks like this:

type KVCache struct {
    Layer0 [...]
    Layer1 [...]
    Layer2 [...]
    ...
}
Enter fullscreen mode Exit fullscreen mode

The cache grows during the entire conversation.

The purpose is straightforward:

Save expensive computation so future tokens can reuse it.

5. Why Doesn't the Model Process the Prompt Again?

Imagine a prompt with 2,000 tokens.

Suppose the model generates 500 output tokens.

One approach would be:

  • Read the 2,000 prompt tokens.
  • Generate one token.
  • Read the same 2,000 prompt tokens again.
  • Generate the next token.
  • Repeat.

That would repeat a large amount of computation.

The KV cache changes the process.

The model reads the prompt once during prefill.

During decode, it reuses the cached information and adds data for each newly generated token.

The expensive work happens once.

6. Wait... How Large Does the KV Cache Become?

This was the next question I asked.

If every prompt token adds data to the cache...

What happens with a one million token prompt?

The answer follows directly from the mental model.

The cache grows with every token.

A rough estimate for a 7B model gives memory usage on the order of hundreds of kilobytes per token.

That leads to numbers like these:

Prompt Length Approximate KV Cache Size
1,000 tokens Hundreds of MB
100,000 tokens Tens of GB
1,000,000 tokens Hundreds of GB

Modern models reduce this memory with techniques such as Grouped Query Attention (GQA). Several query heads share the same Keys and Values, so the cache becomes much smaller.

The important idea is simple.

Long context windows require large amounts of memory because the KV cache keeps growing.

This explains why modern serving systems spend so much effort on KV cache management.

7. A Simple Mental Model

After working through these ideas, I arrived at one picture that I keep in my head.

Prompt
    ↓
Prefill(prompt)
    ↓
KV Cache + First Token
    ↓
Decode(...)
    ↓
Next Token
    ↓
Decode(...)
    ↓
Next Token
    ↓
...
Enter fullscreen mode Exit fullscreen mode

This model answers many basic questions.

  • Why does the model read the prompt before generating text?
  • Why does GPU memory increase with context length?
  • Why does every serving framework discuss the KV cache?
  • Why do serving optimizations focus on prefill and decode?

I still have many questions about attention, batching, scheduling, and parallelism.

Now I have a place to attach those ideas as I learn them.

What part of LLM inference took the longest to click for you? Was it attention, the KV cache, batching, or something else?

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

GitHub logo HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Git Commit



GenAI today is a race car without brakes. It accelerates fast -- you describe something, and large blocks of code appear instantly. But AI agents silently break things: they remove logic, relax constraints, introduce expensive cloud calls, leak credentials, and change behavior -- without telling you. You often find out in production.

git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.

In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen

At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…

View on GitHub

Top comments (0)