When to Use AI Caching, and When Not To

When to Use AI Caching, and When Not To

We start with a simple test, go workload by workload, then cover the pitfalls. Examples refer to ZRouter settings where it helps to be concrete.

The three-question test

Before enabling any response cache (exact or semantic), ask:

1. Do the same requests repeat? If most prompts are unique, a response cache will mostly miss.
2. Is a reused answer still correct? If the right answer depends on time, live data, or the specific user, reuse is risky.
3. Is a reused answer acceptable? Some products need variety (creative writing, brainstorming). Others need consistency (policy answers).

If the answers are yes, yes, and yes, use a response cache. If any is no, skip response caching for that traffic and lean on provider prompt caching instead, which never reuses answers and is safe almost everywhere.

Workload by workload

FAQ bots and support assistants

Use: exact cache, semantic cache, prompt caching.

This is the best case. Questions repeat constantly, answers are stable, and consistency is a feature. Start the semantic threshold high and lower it only when you have evidence. ZRouter's default threshold is 0.92, which is a sensible starting point. Set a TTL that matches how often your underlying content changes. If your help center updates weekly, a one-hour TTL (ZRouter's semantic default, SEMANTIC_CACHE_TTL=3600) is safe.

Retrieval-augmented generation (RAG)

Use: exact cache, prompt caching, embedding caching. Semantic cache with care.

In RAG, the answer depends on the retrieved documents as much as on the question. An exact cache is safe because the retrieved context is part of the request body, so a document change produces a new key. A semantic cache that matches only on the question could return an answer built from old documents. ZRouter reduces this risk because only the final user message is matched by meaning, while the rest of the request, including injected context in other messages and a window of recent conversation (SEMANTIC_CACHE_MAX_CONV_MESSAGES, default 3), still has to match. Even so, keep TTLs short when your corpus changes often.

Embedding caching pays off when you re-index. Identical chunks hit the exact cache on /v1/embeddings instead of being billed again.

Coding agents and multi-step agents

Use: prompt caching. Avoid response caching.

Agents send long, stable prefixes (instructions, tool schemas, file contents) followed by new, specific steps. Each response must reflect the latest state of the task, so a reused answer is almost always wrong. But the prefix is the perfect target for prompt caching, and on long sessions it is where most of the money goes.

Two things decide whether prompt caching actually hits:

• Prefix stability. Put static content first and changing content last. A timestamp at the top of a system prompt breaks the cache on every request.
• Routing stability. The provider cache lives on a specific provider, model, and often API key. If your router load-balances turn 5 to a different target than turn 4, the cache is cold.

ZRouter handles the second point with session keeping: requests in a detected session (for example from Claude Code, Codex CLI, or OpenCode session headers) stay on the same target and the same provider key.

Chat apps with personal context

Use: prompt caching. Response caching only for non-personal requests.

If answers mention the user's name, account, or history, a shared response cache is a data leak waiting to happen. Make sure any response cache is scoped per user or per account. ZRouter includes the authenticated access scope and the server-derived user path in every exact and semantic cache key. Customer API keys are bound to /accounts/ACCOUNT_ID, and client headers cannot override that binding, so two accounts never share cached answers, even for identical prompts.

Batch and offline jobs

Use: exact cache.

Evaluation runs, classification over a dataset, and nightly summaries often re-run the same inputs after a crash or a code change. An exact cache makes reruns cheap. Since timing does not matter, consider a long TTL for the duration of the job.

Creative and generative features

Avoid: response caching.

If users click "regenerate" to get a different poem, serving the same poem defeats the feature. Bypass the cache for those requests. In ZRouter, send Cache-Control: no-cache or Cache-Control: no-store on any request that should skip both response cache layers.

Real-time or time-sensitive answers

Avoid: response caching, or use a very short TTL.

"What is the status of my order?", "What is the BTC price?", and "Is the API down?" need live data. Fresh tool calls and retrieval are the right answer here, and prompt caching can still reduce the cost of the static instructions.

Quick reference

Workload	Exact	Semantic	Prompt cache	Embedding cache
FAQ / support	Yes	Yes, high threshold	Yes	Optional
RAG	Yes	Careful, short TTL	Yes	Yes
Coding agents	Rarely useful	No	Yes, most value	No
Personal chat	Only if scoped	Only if scoped	Yes	No
Batch jobs	Yes	Rarely	Yes	Yes
Creative	No	No	Yes	No
Real-time data	No, or short TTL	No	Yes	No

Pitfalls that cause wrong answers

Semantic thresholds that are too loose

At a low threshold, "cancel my subscription" and "renew my subscription" can look similar. The cost of one wrong hit is usually higher than the savings from many right ones. Start strict, monitor wrong-hit reports, and loosen slowly.

Ignoring parameters

Two requests with the same prompt but different models, temperatures, tools, or response formats are different requests. A cache that keys only on the prompt text will serve a JSON-mode answer to a plain-text request. ZRouter's semantic keys include all top-level request options and provider extensions, and its exact keys include the resolved provider and model, so a change in any of them is a miss.

Caching across policy changes

If you add a guardrail that redacts personal data, cached responses from before the change were never checked by it. ZRouter includes the resolved guardrail chain in the exact cache key. Editing, adding, or removing a guardrail step produces a miss rather than replaying a response the new policy never saw.

Caching failures and fallbacks

Never cache errors. Be careful about caching answers from a fallback model under the primary model's key. ZRouter does not cache failed responses, and it does not cache responses served through failover.

Leaking data between tenants

The most serious pitfall. Any shared response cache must key on the tenant. If you run multi-tenant infrastructure, verify this before enabling caching, not after. Also check how bring-your-own-key traffic is handled. In ZRouter, BYOK requests bypass the shared response cache entirely.

Stale answers after content changes

Pick a TTL that matches how often the truth changes. When you ship a big content update, expect old entries to live until they expire, or clear the cache.

Unstable prompt prefixes

For prompt caching, small changes at the start of a prompt (a date, a random ID, a reordered tool list) silently disable the cache. If your prompt-cache hit rate is near zero, look at the first few hundred tokens of consecutive requests and diff them.

Key takeaways

• Use response caching for repetitive, stable, non-personal traffic.
• Use prompt caching almost everywhere, and especially for agents and long conversations.
• Scope every response cache to the tenant and include every parameter that changes the answer in the key.
• Give users and developers a way to bypass the cache per request.

Next in the series: [Part 4: Where to Put Your AI Cache]