What Is AI Caching? A Practical Guide to the Four Layers

What Is AI Caching? A Practical Guide to the Four Layers

Every call to a large language model costs money and time. You pay for each input token and each output token, and you wait while the model reads your prompt and writes its answer. In a busy application, a large share of those calls are repeats: the same help-center question, the same system prompt, the same 20-page document sent with every turn of a conversation.

AI caching is the practice of noticing that work and not paying for it twice.

This is the first post in a four-part series. Here we cover what AI caching is and the different layers it can happen at. Part 2 covers the benefits in numbers, Part 3 covers when caching helps and when it hurts, and Part 4 covers where to put each layer in your stack. Throughout the series we use ZRouter, an open source, OpenAI-compatible AI gateway, as a working reference, because it implements every layer described here in one place.

Caching, but for models

Classic web caching is simple. A request for /logo.png always returns the same bytes, so you store them and serve the copy next time.

AI requests are messier for three reasons:

1. Outputs are not deterministic. Ask a model the same question twice and you may get two different wordings. Caching an answer means choosing one and reusing it.
2. Inputs are large and mostly repeated. A chat request carries the full conversation history, a long system prompt, tool definitions, and maybe retrieved documents. The new part of each request is often a few dozen tokens at the end.
3. The same question has many spellings. "How do I reset my password?" and "password reset steps?" mean the same thing but share almost no characters.

Because of this, AI caching is not one technique. It is a set of layers, each catching a different kind of repetition.

Layer 1: Exact response caching

What it is: Store the complete response to a request. When an identical request arrives, return the stored response without calling the model.

What counts as identical: The request body after normalization, plus anything that changes the meaning of the request: which model it resolved to, which account sent it, and which policies ran on it.

What you save: The whole call. No input tokens, no output tokens, and the response comes back in milliseconds instead of seconds.

Example: A documentation assistant receives "What are your API rate limits?" two hundred times a day, sent by a widget that builds the exact same request every time. After the first call, the other 199 are free.

In ZRouter, the exact cache is backed by Redis and covers /v1/chat/completions, /v1/responses, /v1/messages, and /v1/embeddings. A hit is marked with a response header so you can see it:

X-Cache: HIT (exact)

Two details make the ZRouter version more useful than a naive "hash the body" approach:

• Canonicalization. JSON key order and whitespace are normalized before hashing, so two clients that format the same request differently still share one entry.
• Request coalescing. If ten identical requests arrive at the same moment and none is cached yet, only one goes to the provider. The other nine wait and replay its answer. This prevents a burst of traffic from turning into a burst of identical paid calls.

Streaming is handled too. A streaming miss stores the raw server-sent event bytes, and a streaming hit replays them as text/event-stream, so your client code does not change.

Layer 2: Semantic caching

What it is: Store responses along with an embedding (a vector representing meaning) of the prompt. When a new prompt arrives, embed it, search for a stored prompt whose vector is close enough, and if one is found, return its response.

What counts as a match: Similarity above a threshold, usually expressed as cosine similarity between 0 and 1. A high threshold such as 0.92 means "almost the same question". A lower threshold catches more paraphrases but risks wrong answers.

What you save: The model call, at the price of one embedding call and one vector search. Embeddings are typically far cheaper than generation, so the trade is usually good for repeated question patterns.

Example: A support bot gets "how do I reset my password", "I forgot my password, what now?", and "password reset link not working?" The first two are the same need and can share an answer. The third is a different problem, and a well-tuned threshold keeps it separate.

ZRouter's semantic layer uses an embedding provider you choose plus a vector store: Qdrant, pgvector, Pinecone, or Weaviate. A hit is labelled:

X-Cache: HIT (semantic)

ZRouter is deliberately conservative about what it matches on. Only the final user message is compared by meaning. Everything else (model, temperature, tools, other request options, and a limited window of conversation context) must match exactly as part of the key. That way a semantic hit never crosses from one model or configuration to another. It also never applies semantic matching to /v1/embeddings, because an embedding has to represent the exact text it was requested for.

Layer 3: Provider prompt caching

What it is: The model provider keeps the processed form of a long, repeated prompt prefix in memory. When your next request starts with the same prefix, the provider skips reprocessing it and bills those tokens at a discount.

What counts as a match: A shared prefix. The system prompt, tool definitions, and earlier conversation turns must be byte-identical and appear first. Anything after the first difference is processed normally.

What you save: Part of the input cost and part of the time to first token. Output is still generated fresh, because the question at the end is new.

Example: A coding agent sends a 30,000-token system prompt and project context on every turn. With prompt caching, turns two through fifty reuse that prefix. The new tokens at the end are billed normally, and the cached prefix is billed at the provider's reduced cached-input rate.

This layer is different from the first two. The response is not reused at all. You still get a new, fresh answer for every request. What gets reused is the model's work of reading the prompt.

Each provider exposes it differently. Some cache automatically, some need a cache key, some need explicit breakpoints, and some need you to create a cached-content object ahead of time. ZRouter handles this with a provider prompt-cache planner that is on by default. After it knows which provider and model will serve a request, it adds the right directive:

• OpenAI: a stable prompt_cache_key, plus explicit breakpoints on models that support them.
• Anthropic: automatic ephemeral caching.
• Amazon Bedrock (Converse): a cache point after the stable prefix.
• Google Gemini (AI Studio): a cached-content object, created only after the same prefix is seen again within five minutes.

If your client already sends its own cache directives, ZRouter leaves them alone.

Layer 4: Embedding caching

What it is: Store the vector for a piece of text so you do not embed the same text twice.

What you save: Embedding calls. Each one is cheap, but RAG pipelines and semantic caches make a lot of them.

ZRouter uses this in two ways. Requests to /v1/embeddings participate in the exact cache, so re-indexing the same documents does not re-bill you. Inside the semantic cache, identical lookup text is reused in memory for five minutes within the same account and embedder, and concurrent identical lookups share a single embedding call.

A fifth layer you do not control: the KV cache

Inside the model server, every transformer keeps a key-value (KV) cache of the tokens it has already processed in the current generation. It is the reason the second word of an answer comes out faster than the first. Provider prompt caching is essentially that KV cache, kept alive between your requests. You do not configure it directly, but it explains why prompt structure matters so much, which we cover in Part 3.

The layers side by side

Layer	What is reused	Match rule	Saves	Answer is fresh?
Exact response cache	Full response	Identical request	Entire call	No
Semantic cache	Full response	Similar final prompt, same everything else	Entire call, minus one embedding	No
Provider prompt cache	Processed prompt prefix	Identical leading tokens	Part of input cost and latency	Yes
Embedding cache	Vectors	Identical text	Embedding calls	Not applicable

The layers stack. A request can miss the exact cache, miss the semantic cache, and still get a prompt-cache discount from the provider. In ZRouter, all of them run in one gateway, in that order, after routing, workflows, and guardrails have produced the final request.

Key takeaways

• AI caching is a stack of layers, not a single switch.
• Response caches (exact and semantic) skip the model entirely but reuse an old answer.
• Prompt caching keeps every answer fresh while cutting the cost of the repeated prefix.
• The right mix depends on how repetitive your traffic is and how much freshness matters.

Next in the series: Part 2: The Real Benefits of AI Caching, where we put numbers on cost, latency, rate limits, and reliability.