DEV Community

Yaseen Khatib
Yaseen Khatib

Posted on Originally published at yaseenkhatib.streamerosai.com

Semantic Caching That Actually Hits: Matching Meaning at the Edge

#llm #semantic #vector

[ EXECUTIVE TEARDOWN // TL;DR ]

  • String-keyed caches miss because no two users phrase a question the same way; semantic caching matches on meaning.
  • Reuse the query embedding you already compute to look up previously answered questions in a small vector index.
  • The similarity threshold is a risk decision, not a constant — keep it conservative for grounded agents.
  • Tag cache entries with the corpus version and invalidate on re-index to avoid serving stale answers.

Running the streamerOS support agent taught me that support traffic is gloriously repetitive. "How do I connect my OBS?", "how do I link OBS", "OBS setup help" — three phrasings of one question, and a naive RAG pipeline runs the full retrieve-and-generate gauntlet for every single one. Each is a model call I paid for and a second of latency the user waited through, to produce an answer already produced a hundred times. Semantic caching fixes this by recognising the question, not the string.

Why a normal cache misses

A key-value cache keyed on the raw query string is useless here, because no two users phrase a question identically. The cache hit rate hovers near zero. What you actually want to match on is meaning — and you already have the machinery to measure meaning, because your RAG pipeline embeds the query anyway. A semantic cache reuses that embedding to ask a different question: have I already answered something that means this?

The cheapest model call is the one you never make. Above the threshold, the answer is already on the shelf.

The mechanism

Embed the incoming query — you were going to anyway. Before doing any retrieval or generation, search a small vector index of previously answered queries. If the nearest neighbour sits above a similarity threshold, return its stored answer and stop. If not, run the full pipeline, then write the new query embedding and its answer back to the cache. The cache learns the shape of your traffic over time, and the hot questions go nearly free.

semantic-cache.ts

// check meaning, not string equality
const qVec = await embed(query);
const [near] = await cache.query({ vector: qVec, topK: 1 });

if (near && near.score >= 0.95) {
  return near.metadata.answer;        // HIT — no retrieval, no LLM
}

const answer = await runRagPipeline(query, qVec);
await cache.upsert({ vector: qVec, metadata: { answer } });  // learn it
return answer;
Enter fullscreen mode Exit fullscreen mode

The threshold is a product decision

The similarity floor is the one dial that matters, and it is not an engineering constant — it is a risk choice. Set it high (0.97+) and you only ever reuse answers to near-identical questions; safe, lower hit rate. Set it lower and you catch more paraphrases but risk serving a confidently adjacent answer to a subtly different question. For a grounded support agent I keep it conservative, because a wrong cache hit undoes the entire zero-hallucination guarantee.

Invalidation is the catch

A cached answer is a snapshot of the documentation at the moment it was generated. When the knowledge base changes, stale entries become a liability — they will happily serve last month's instructions. The clean fix is to tag every cache entry with the corpus version it was derived from and drop the whole cache on re-index. The cache is cheap to rebuild; serving a confidently outdated answer is not.

Semantic caching is the rare optimisation that improves cost and latency at once — you are not making the model faster, you are skipping it entirely for questions you have already answered.

This sits in front of the edge-native pipeline and respects the grounding contract — a cache hit is only valid if the original answer was grounded. It is the kind of trade-off I weigh on every system I ship: the cheapest, safest win is usually the work you can avoid doing at all, and knowing where that line sits is most of the job.

~/keep-reading

YK

Yaseen Khatib · AI Architect

Ships autonomous AI products solo — five in the last twelve months. More about Yaseen →

Need an engineer who can build this?

I'm Yaseen Khatib — a Senior Full-Stack AI Engineer (MERN + TypeScript) who ships production AI systems solo. Open to senior and lead roles, remote or on-site.

Get in touch →See what I've shipped

Originally published at yaseenkhatib.streamerosai.com/blog/semantic-caching-edge-rag/.

Top comments (0)