Intro
There's a pitch behind every new AI-powered feature: it understands anything a user throws at it. Open-ended, flexible, genuinely intelligent.
The pitch is half true. What it leaves out is that in most production systems, "anything a user throws at it" collapses into the same twenty questions asked twenty different ways. A support bot doesn't get asked infinite variations of physics, it gets asked "what's your return policy," "how do I get a refund," and "can I send this back" on a loop, forever.
That gap, between the imagined variety of user queries and the actual repetition underneath it, is where the token bill lives.
The naive approach: every request hits the model
Say a team ships a support assistant. Every user message goes straight to a frontier model: embed the question, retrieve context, generate an answer, return it. It works. It's also expensive in a very specific way, identical intent, paid for from scratch, every single time.
(Illustrative, not a specific case, but recognizable to anyone who's watched an LLM API bill by day one versus day thirty.)
Exact-match caching almost works, then doesn't
The obvious fix is a cache: hash the incoming query, check for a hit, skip the model call if found. This works exactly once, for the exact same string. "How do I reset my password" and "forgot my password, help" are the same question and two different cache keys. Exact-match caching is a solution for a problem language doesn't actually have, users asking things identically.
Embeddings turn paraphrase into a solvable problem
Semantic caching replaces the hash with a vector. Embed the incoming query, run a nearest-neighbor search against previously answered queries, and if the closest match clears a similarity threshold, serve the cached response instead of calling the model at all. Below the threshold, call the model as normal and add the new query/response pair to the cache for next time.
This is the actual mechanism at work: not "the AI understood the question was similar," but a distance calculation in vector space with a cutoff you chose.
A stale cache is worse than no cache
Here's the failure mode nobody budgets for. A support answer gets cached in March. The return policy changes in April. The cache doesn't know that, it just has a vector close enough to keep matching, and it will confidently serve outdated information for as long as the entry lives. A cache without an expiration policy isn't a cost optimization, it's a bug that pays for itself to keep running.
Threshold tuning is the whole game
Set the similarity threshold too loose and you'll serve March's return policy answer to an April refund question, because the two "look" adjacent in vector space even though the real answer changed. Set it too tight and you get single-digit hit rates, because no two users phrase anything identically. There's no universal number here, only a number you have to measure against your own traffic.
Build it yourself, or don't
If you're at the scale where cache hit rate is a line item on your infra bill, and your domain has stable, well-defined intents (support, FAQ, internal tooling), rolling your own semantic cache with something like pgvector or Redis is a weekend project, not a research problem. If your queries are genuinely open-ended and low-repeat (creative generation, one-off analysis), you're optimizing for a hit rate that doesn't exist, and the caching layer is wasted engineering effort.
Semantic caching isn't really an AI technique. It's ordinary distributed-systems caching (TTLs, invalidation, cache keys) with an embedding standing in for the key. The AI part was never the hard part.
We ran into a version of this question building Cyclopt Companion's analyzers: a one-line diff shouldn't necessarily trigger a full re-analysis from scratch, and figuring out what counts as "close enough to skip" turned out to be the actual engineering problem, not the analysis itself.
What's your current cache hit rate on user-facing LLM calls, and have you actually measured it, or is it a guess?
Top comments (1)
The staleness point is the one that quietly ends most semantic-cache projects. Threshold tuning at least fails loudly-ish — you can measure false-hit rate on a labeled set. Staleness fails on a delay: the March answer keeps matching in April and the vector distance has no idea the ground truth moved underneath it. The fix that actually held for us wasn't a TTL (too blunt — evicts fresh entries, keeps stale ones that happen to be young), it was tying eviction to the source: when the return-policy doc changes, invalidate every cache entry whose retrieval touched that doc. Cache key includes a content hash of the sources, not just the query vector.
The other thing worth caching is refusals — "we don't support that" is high-repetition and cheap to serve, but people forget to cache the negative path. How are you measuring the false-positive rate on hits in practice? A shadow call to the model on a sampled % of cache hits is the only honest way I've found to know your threshold is still right.