Large language models are expensive to run and slow to respond compared with almost any other component in a modern application stack. A single completion can cost orders of magnitude more than a database query and take a thousand times longer. As soon as an application serves real traffic, patterns emerge: the same questions recur, the same prompts repeat across users, and a surprising share of requests ask for something the system has already answered. Caching turns that repetition into savings. This article explains the economics and architecture of caching at the AI gateway layer, why the gateway is the right place to do it, and the pitfalls to watch for.
The core idea is simple. An AI gateway sits in the request path between your applications and your model providers. Because every call passes through it, the gateway can recognise a repeated or near-duplicate request and return a stored response without making a new model call. That single capability cuts both cost and latency at once, and it does so centrally, so individual teams do not each have to build it.
The economics of a cache hit
To see why caching matters, consider what a cache hit avoids. A cache miss follows the full path: your application sends a request, the gateway forwards it to a provider, the model runs a forward pass, tokens stream back, and you are billed for every input and output token. A cache hit short-circuits all of that. The response is read from a store and returned. You pay nothing to the model provider for that request, and the user waits only as long as the lookup takes rather than as long as generation takes.
Two different costs fall at once:
- Spend. Each served-from-cache request is a model call you did not pay for. Across a workload with meaningful repetition, avoided calls accumulate into a real reduction in provider bills.
- Latency. A stored response returns far faster than a fresh generation. For interactive products, that difference is felt directly by users; for batch and agentic workloads, it compounds across many steps.
The lever that governs both is hit rate: the proportion of requests served from cache. A workload with high repetition and a well-tuned cache can serve a large share of traffic without touching a model. A workload of mostly unique, personalised requests will see little benefit. The economics follow directly from how repetitive your traffic actually is, which is why measuring hit rate matters more than any headline promise.
Why the gateway is the right layer
You could implement caching inside each application, but doing it at the gateway is better for several structural reasons.
It already sees everything
The gateway is the one component every request flows through. That makes it the natural place for any logic that needs a complete view of traffic. A cache is only as good as the share of requests it can observe; a cache that sees all traffic can match a request from one application against a response generated for another, where that is safe to do. Scattering caches across services fragments that view and duplicates effort.
It centralises the hard parts
Correct caching requires careful handling of keys, expiry, invalidation, and isolation. Implementing that once, well, at the gateway is far more reliable than reimplementing it in every service in every language your organisation uses. Teams keep making ordinary model calls; the caching behaviour lives in shared infrastructure.
It sits with related controls
Caching is one of a family of cross-cutting concerns the gateway handles, alongside routing, budgets, guardrails, and observability. These interact. The same layer that enforces a budget can reduce the spend that budget tracks; the same layer that records observability data can report cache hit rate as a first-class metric. Keeping caching beside these controls means they share one consistent view of traffic.
Applications reach the gateway through an ordinary OpenAI-compatible client, so adopting it does not change application code:
from openai import OpenAI
client = OpenAI(
api_key="your_truefoundry_api_key",
base_url="https://gateway.truefoundry.ai",
)
response = client.chat.completions.create(
model="openai/gpt-4o",
messages=[{"role": "user", "content": "Summarise our return policy."}],
)
From the application's perspective nothing special happens; the gateway decides per request whether to serve a cached answer or forward to the model.
Configuring a cache: an illustrative example
Caching behaviour is usually expressed as configuration rather than code. The snippet below is illustrative and generic, not a specific product API; it shows the kinds of knobs a cache exposes and how they map to the concepts above.
Illustrative cache configuration (generic, not a real API)
cache:
enabled: true
mode: semantic # or "exact" for byte-identical matching
similarity_threshold: 0.92 # how close a match must be to reuse (semantic)
ttl: 3600 # seconds before an entry expires
scope: per_tenant # isolate entries by tenant to avoid leakage
exclude:
- personalised
- tool_calls
Each field corresponds to a decision worth understanding. mode chooses between exact-match caching, which reuses an answer only when the input is identical, and semantic caching, which reuses when a new prompt is close enough in meaning to a stored one. similarity_threshold governs how close "close enough" is for semantic matching and trades precision against recall. ttl bounds staleness. scope controls isolation. exclude keeps unsafe categories out of the cache entirely.
Cache isolation and safety
A cache that serves the wrong person's answer is worse than no cache at all. The central safety concern is isolation: making sure a stored response is only ever returned to a request that is genuinely entitled to the same answer.
- Per-user and per-tenant partitioning. In multi-tenant systems, cache lookups must be scoped so one tenant can never receive another tenant's cached content. This is a boundary, not an optimisation. -** Excluding personal data.** Responses that embed a specific user's data should not sit in a shared cache. Either exclude them or confine them to that user's private scope.
- Respecting context. If a system prompt, injected context, or model parameter changes the correct answer, it must be part of what the cache keys on, or you will serve a response generated under different conditions.
Because the gateway owns identity and routing for every request, it is well placed to enforce these boundaries consistently rather than leaving each application to get isolation right on its own.
Invalidation: keeping answers fresh
Every cached answer is a bet that the underlying truth has not changed. Invalidation is how you manage that bet. Two mechanisms work together:
- Time to live. A TTL sets a ceiling on staleness. Once an entry expires, the next matching request goes to the model and refreshes the stored answer. Short TTLs suit volatile information; longer TTLs suit stable facts.
- Explicit eviction. When a source changes, such as an updated policy or a corrected document, you need to purge affected entries immediately rather than wait for expiry. Namespace versioning (bumping a version tag so old entries become unreachable) and tag-based eviction (removing all entries derived from a given source) are the usual tools.
The right balance depends on your tolerance for staleness. A support bot answering stable product questions can use long TTLs; a system quoting anything that moves should keep them short and lean on explicit eviction.
Interaction with routing and budgets
Caching does not operate in isolation at the gateway; it interacts with the other controls, and understanding that interaction prevents surprises.
With routing, the cache sits logically in front of the routing decision: if a request can be served from cache, there is no need to choose a provider or model for it at all. That ordering is what delivers the latency win, since the fastest model call is the one you never make. With budgets, caching changes the shape of spend. A cached request consumes no provider tokens, so a well-populated cache can keep a workload within budget that would otherwise exceed it. Reporting should make this visible, attributing savings to the cache so teams understand where their spend did and did not go. With observability, hit rate becomes a metric to track over time, because it can drift as traffic patterns change, and a falling hit rate is an early signal that your cache or threshold needs attention.
Pitfalls to watch for
- Caching what should not be cached. Personalised, time-sensitive, tool-calling, and high-stakes outputs do not belong in a shared cache. Decide exclusions deliberately rather than caching everything by default.
- A threshold set too loose. For semantic caching, an aggressive similarity threshold inflates hit rate while serving answers that only looked similar. Watch answer quality, not just hit rate.
- Stale answers from missing invalidation. Relying on TTL alone means updates take effect only after expiry. Pair TTL with explicit eviction for anything that can change underneath you.
- Isolation gaps. A cache that is not correctly partitioned can leak one user's content to another. Treat isolation as a correctness requirement.
- Measuring the wrong thing. A high hit rate paired with degraded quality is a loss disguised as a win. Track hit rate, latency, savings, and quality together.
Key Takeaways
- A cache hit avoids a full model call, cutting both provider spend and user-facing latency at the same time.
- Hit rate, driven by how repetitive your traffic is, determines how much caching actually saves, so measure it.
- The gateway is the right layer because it already sees every request, centralises the hard parts of caching, and sits beside routing, budgets, guardrails, and observability.
- Cache configuration exposes clear knobs: exact versus semantic mode, a similarity threshold, a TTL, an isolation scope, and exclusions.
- Isolation per user and tenant is a safety requirement, not an optimisation; never serve one party's cached answer to another.
- Combine TTL with explicit invalidation so updated sources take effect promptly.
- Avoid caching personalised, time-sensitive, tool-calling, or high-stakes outputs, and never read hit rate without reading quality alongside it.
Top comments (0)