Where to Put Your AI Cache: Client, Gateway, or Provider

Where to Put Your AI Cache: Client, Gateway, or Provider

There are three places: inside your application, at an AI gateway between your apps and the providers, and at the provider itself. Each has a role. This post compares them and then walks through a complete setup using ZRouter, an open source AI gateway that runs every caching layer in one place.

Option 1: Cache in the application

You add a cache library or a Redis call around your LLM client code.

Good for:

• A single app with a narrow use case, such as caching one expensive summary per document.
• Business-level caching, where you know something the model call does not. For example, "this product description is unchanged, so reuse yesterday's summary."

Weak at:

• Duplication. Every service, language, and team writes its own cache with its own bugs.
• Correctness details. Key design, streaming replay, request coalescing, tenant isolation, and invalidation on policy changes are easy to get subtly wrong.
• Provider prompt caching. Each provider has its own directives. Your app code ends up full of provider-specific fields that break when you switch models.
• Visibility. Hit rates are scattered across services.

Option 2: Cache at the provider

Provider prompt caching runs on the provider's servers. You mostly need to structure prompts well and, for some providers, send a directive.

Good for:

• Long, stable prefixes. It is the only layer that can reuse the model's own processing.
• Keeping every answer fresh.

Weak at:

• It never skips the call. You still pay for output and for the new part of the input.
• Every provider works differently: automatic caching, cache keys, breakpoints, or explicit cached-content objects with their own lifetimes.
• It depends on routing. Spreading one conversation across providers, models, or API keys can leave the cache cold.

Option 3: Cache at the gateway

An AI gateway is a proxy that sits between your applications and model providers. All LLM traffic passes through it, which makes it the natural home for shared concerns: routing, failover, budgets, guardrails, logging, and caching.

Good for:

• One cache for every app. Any client that speaks the OpenAI API benefits without code changes.
• Correct keys. The gateway knows the authenticated tenant, the resolved provider and model, and which policies ran, so it can key on all of them.
• Provider prompt caching without provider code. The gateway knows the final provider after routing and can add the right directive for it.
• Routing that respects caches. The gateway can keep a session on the same target and key so provider caches stay warm.
• One place to measure. Hit rates and cost avoided appear in one dashboard.

Weak at:

• Business-level knowledge that only your app has. Keep that in the app.

The recommended layout

For most teams the answer is a combination:

1. Gateway: exact cache, semantic cache where it fits, prompt-cache planning, and session stickiness.
2. Provider: prompt caching, driven by the gateway.
3. Application: only domain-specific caching, plus stable prompt structure (static content first).

Setting it up in ZRouter

The examples below show configuration only. Keep real API keys and connection strings in your secret manager or environment, never in committed files.

Step 1: Provider prompt caching (already on)

ZRouter's provider prompt-cache planner is enabled by default. To make that explicit in your deployment:

PROVIDER_PROMPT_CACHE_PLANNER_ENABLED=true

For Gemini, you can tune the lifetime of the cached-content objects ZRouter creates. The default is 300 seconds, and values from 30 to 3600 are accepted:

GEMINI_PROMPT_CACHE_TTL_SECONDS=300

If your client already sends its own cache directives (for example Anthropic cache_control blocks), ZRouter passes them through unchanged. To turn off automatic planning but keep pass-through, set the flag to false.

Step 2: Exact response cache

Point ZRouter at Redis in config.yaml:

cache:
  response:
    simple:
      redis:
        url: redis://localhost:6379
        ttl: 3600

Or with environment variables only:

RESPONSE_CACHE_SIMPLE_ENABLED=true
REDIS_URL=redis://localhost:6379
REDIS_TTL_RESPONSES=3600

Step 3: Semantic cache (optional)

Add an embedder and a vector store. Qdrant is shown here. pgvector, Pinecone, and Weaviate are also supported.

cache:
  response:
    semantic:
      enabled: true
      embedder:
        provider: openai
        model: text-embedding-3-small
      vector_store:
        type: qdrant
        qdrant:
          url: http://localhost:6333
          collection: aigateway_semantic

Useful tuning variables:

SEMANTIC_CACHE_THRESHOLD=0.92            # similarity required for a hit
SEMANTIC_CACHE_TTL=3600                  # seconds an entry lives
SEMANTIC_CACHE_MAX_CONV_MESSAGES=3       # recent conversation included in the key
SEMANTIC_CACHE_EXCLUDE_SYSTEM_PROMPT=false

If you use pgvector or Pinecone, make sure the configured dimension matches your embedding model's output size (1536 for text-embedding-3-small).

When both layers are enabled, a semantic hit is also remembered in Redis for that exact request. The next identical request skips the embedding and the vector search and is still reported as HIT (semantic).

Step 4: Send traffic and check the headers

Any OpenAI-compatible client works. Send the same request twice:

curl -i https://your-gateway/v1/chat/completions \
  -H "Authorization: Bearer $ZROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"openai/gpt-4o-mini","messages":[{"role":"user","content":"What are your API rate limits?"}]}'

The second response includes:

X-Cache: HIT (exact)

A paraphrased question, with the semantic layer enabled, can return X-Cache: HIT (semantic).

Step 5: Bypass when you need fresh output

For "regenerate" buttons, live data, or debugging, skip both response cache layers for one request:

-H "Cache-Control: no-cache"

Use no-store when the response should not be written to the cache either.

Step 6: Keep sessions sticky for agents

If you run coding agents through ZRouter, session keeping works with zero configuration. Known session headers such as X-Session-Id, X-Claude-Code-Session-Id, and Codex's Session-Id keep a session on the same load-balanced target and the same provider key, so provider prompt caches stay warm for the whole task.

Step 7: Measure

Cache hits are recorded as usage entries with cache_type set to exact or semantic:

# Hits only, for one team
curl "https://your-gateway/admin/usage/summary?user_path=/team/alpha&cache_mode=cached" \
  -H "Authorization: Bearer $AIGATEWAY_MASTER_KEY"

Use /admin/cache/overview for a cache-only summary. Default usage reports exclude hits, so cost totals keep matching your provider invoice.

What ZRouter handles so you do not have to

A quick checklist of the details covered in Part 3, and how the gateway deals with them:

Concern	How ZRouter handles it
Tenant isolation	Keys include the auth scope and account-bound user path
Parameter changes	Exact keys include the final body and resolved model; semantic keys include all request options
Policy changes	The guardrail chain is part of the exact cache key
Formatting differences	JSON is canonicalized before hashing
Traffic spikes	Concurrent identical misses are coalesced per process
Streaming	Raw event streams are stored and replayed
Failover answers	Not cached under the primary's key
BYOK traffic	Bypasses the shared response cache
Provider differences	One planner adds the right prompt-cache directive per provider

Wrapping up the series

AI caching is one of the few optimizations that cuts cost and latency at the same time. The key ideas from all four parts:

1. Know the layers. Exact and semantic caches reuse answers. Prompt caching reuses the model's reading of your prompt. Embedding caching reuses vectors.
2. Measure the benefit. Track hit rate and cost avoided per layer.
3. Match the layer to the workload. Response caches for repetitive, stable traffic. Prompt caching nearly everywhere, especially for agents.
4. Put shared caching at the gateway. It sees every request, knows the tenant and the final model, and keeps provider caches warm.

If you want to try it, ZRouter is open source under the MIT license and speaks the OpenAI API, so you can point an existing client at it and turn on each layer one step at a time.