KV Caching in LLMs: How It Speeds Up Text Generation

#llm #ai

Nowadays, running your own LLM can be very handy in some situations, so learning more deeply about it would be beneficial.

One such concept is KV Caching.

In this article, I will show what KV Caching is.

What is KV Caching?

KV caching, which is short for Key-Value Caching, is a key optimization technique used in LLMs.

The highlight of this is that it makes text generation much faster.

The problem that KV caching solves

LLMs generate text one token at a time. (A token is roughly a word or part of a word.)

They use a part of the transformer architecture called the attention.

It is basically where the model looks back at all the previous tokens to decide the next one.

Without KV Caching

For each new token, the model would recompute attention over the entire history from scratch.
As the conversation grows, it gets slower and slower.

With KV Caching

The model caches the key and value vectors from the previous tokens.
When a new token is generated, it only computes the new keys and values for the latest token.
The previous ones are cached, so they are reused.
As a result, each step is much faster.

Pros and Cons

Pros
- Much faster generation
Cons
- The cache will use extra memory (VRAM on GPU or RAM on CPU)
- It grows with context length (prompt + generated text)

Wrapping up

Hope you got a better understanding of what KV caching means. If you are a person who is willing to explore more and improve your workflow, I have a suggestion for you.

It’s free, open-source, and built with developers in mind.