[ EXECUTIVE TEARDOWN // TL;DR ]
- String-keyed caches miss because no two users phrase a question the same way; semantic caching matches on meaning.
- Reuse the query embedding you already compute to look up previously answered questions in a small vector index.
- The similarity threshold is a risk decision, not a constant — keep it conservative for grounded agents.
- Tag cache entries with the corpus version and invalidate on re-index to avoid serving stale answers.
Running the streamerOS support agent taught me that support traffic is gloriously repetitive. "How do I connect my OBS?", "how do I link OBS", "OBS setup help" — three phrasings of one question, and a naive RAG pipeline runs the full retrieve-and-generate gauntlet for every single one. Each is a model call I paid for and a second of latency the user waited through, to produce an answer already produced a hundred times. Semantic caching fixes this by recognising the question, not the string.
Why a normal cache misses
A key-value cache keyed on the raw query string is useless here, because no two users phrase a question identically. The cache hit rate hovers near zero. What you actually want to match on is meaning — and you already have the machinery to measure meaning, because your RAG pipeline embeds the query anyway. A semantic cache reuses that embedding to ask a different question: have I already answered something that means this?
The cheapest model call is the one you never make. Above the threshold, the answer is already on the shelf.
The mechanism
Embed the incoming query — you were going to anyway. Before doing any retrieval or generation, search a small vector index of previously answered queries. If the nearest neighbour sits above a similarity threshold, return its stored answer and stop. If not, run the full pipeline, then write the new query embedding and its answer back to the cache. The cache learns the shape of your traffic over time, and the hot questions go nearly free.
semantic-cache.ts
// check meaning, not string equality
const qVec = await embed(query);
const [near] = await cache.query({ vector: qVec, topK: 1 });
if (near && near.score >= 0.95) {
return near.metadata.answer; // HIT — no retrieval, no LLM
}
const answer = await runRagPipeline(query, qVec);
await cache.upsert({ vector: qVec, metadata: { answer } }); // learn it
return answer;
The threshold is a product decision
The similarity floor is the one dial that matters, and it is not an engineering constant — it is a risk choice. Set it high (0.97+) and you only ever reuse answers to near-identical questions; safe, lower hit rate. Set it lower and you catch more paraphrases but risk serving a confidently adjacent answer to a subtly different question. For a grounded support agent I keep it conservative, because a wrong cache hit undoes the entire zero-hallucination guarantee.
Invalidation is the catch
A cached answer is a snapshot of the documentation at the moment it was generated. When the knowledge base changes, stale entries become a liability — they will happily serve last month's instructions. The clean fix is to tag every cache entry with the corpus version it was derived from and drop the whole cache on re-index. The cache is cheap to rebuild; serving a confidently outdated answer is not.
Semantic caching is the rare optimisation that improves cost and latency at once — you are not making the model faster, you are skipping it entirely for questions you have already answered.
This sits in front of the edge-native pipeline and respects the grounding contract — a cache hit is only valid if the original answer was grounded. It is the kind of trade-off I weigh on every system I ship: the cheapest, safest win is usually the work you can avoid doing at all, and knowing where that line sits is most of the job.
~/keep-reading
- 7 min readMongoDB Aggregation Pipelines: Stage Order Is the WinProfiling a clinical API taught me MongoDB aggregation pipeline optimization lives or dies on stage order: $match first on indexes, $lookup join performance, then explain.
- 8 min readI Built a RAG Chatbot on Cloudflare Workers. It Broke 3 WaysA RAG chatbot on Cloudflare Workers, Workers AI and Vectorize, built in one night, then broken by a deprecated model, a free-tier cap and a stale index.
- 7 min readAI Engineer Portfolio Examples: What Reviewers VerifyFour kinds of AI engineer portfolio pieces that let a reviewer verify production experience in minutes: a live grounded assistant, a product with a readable trust boundary, a measured result, and the operating model. With this site as the worked example and its failures included.
YK
Yaseen Khatib · Full Stack AI Developer
Ships autonomous AI products solo — five in the last twelve months. More about Yaseen →
Want this built for your business?
I build what this post describes for clients — RAG chatbots on your own documents, AI added to systems you already run, custom software — at fixed prices you see before we start. Hiring instead? I'm open to senior and lead roles.
See services and prices →I'm hiring
Originally published at yk.streamerosai.com/blog/semantic-caching-edge-rag/.
Originally published on my portfolio.
Top comments (0)