DEV Community

Devanshu Biswas
Devanshu Biswas

Posted on

Semantic caching keys on meaning, not the exact string — embed the query, cosine-match the cache, HIT above a threshold

#ai #llm #machinelearning #performance

A plain cache keys on the exact string, so "How do I reset my password?" and "I forgot my password, how can I change it?" look like two different requests and both hit the model — even though the answer is identical. Semantic caching keys on meaning: it embeds the incoming query into a vector, measures cosine similarity against previously answered queries, and if the closest clears a threshold it returns that cached answer — a HIT — instead of calling the model, cutting both cost and latency. I built a live cache that scores every entry as you type. Here's how it works.

Embed the query into a meaning vector

An embedding maps text to a vector so similar meanings land near each other. Real systems call an embedding model; the demo uses a tiny deterministic bag-of-concepts — lowercase, drop stop-words, collapse synonyms to a shared token — enough to make paraphrases overlap with no network.

const SYN = { forgot:'reset', change:'reset', delivery:'shipping', price:'cost' };
function embed(text){
  const v = new Set();
  for (const w of text.toLowerCase().match(/[a-z]{2,}/g) || [])
    if (!STOP.has(w)) v.add(SYN[w] || w);   // synonyms collapse to one concept
  return v;                                  // a sparse meaning vector
}
Enter fullscreen mode Exit fullscreen mode

Cosine similarity, then the threshold gate

Compare two embeddings by the cosine of the angle between them: 1.0 = same meaning, 0 = nothing in common. Embed the query, find the most similar cached entry, and serve it only if its similarity clears the threshold. Above → HIT (no model call); below → MISS (call the model and insert the new Q→A, so the next paraphrase is a hit).

let best = null, bestSim = -1;
for (const e of cache){ const s = cosine(q, e.embedding);
  if (s > bestSim){ bestSim = s; best = e; } }        // nearest neighbour
return bestSim >= threshold ? { hit:true, answer:best.answer } : { hit:false };
Enter fullscreen mode Exit fullscreen mode

The threshold is the whole game

Set it too high and near-identical questions miss, so you overpay. Set it too low and unrelated questions that share a word or two collide, and you confidently serve a wrong cached answer — a false hit. "What is the capital of Germany?" against a cache of support FAQs is the trap. Pick the threshold on a labelled set to maximise hits at zero false hits, and keep a grey band where, when unsure, you pay rather than risk it.

if (sim >= SERVE) return 'HIT';        // 0.83: confidently reuse
if (sim <  CALL)  return 'MISS';       // 0.60: clearly new -> call model
return 'MISS_SAFE';                    // grey zone: pay rather than risk a wrong answer
Enter fullscreen mode Exit fullscreen mode

The payoff, in numbers

Each avoided call is real money and latency handed back. The demo's toy model books roughly $0.0021, ~900 ms and ~380 tokens per hit; a steady 60% hit rate is about 60% off the inference bill for repeated questions, and the hit rate only climbs as the cache warms up.

if (hit){ hits++; costSaved += 0.0021; msSaved += 900; }
const hitRate = hits / queries;               // rises as traffic repeats
Enter fullscreen mode Exit fullscreen mode

vs prompt caching, plus keeping it fresh

Because it matches on meaning, it catches paraphrases — that's what separates it from prompt caching (Day 31), which only reuses the prefill/KV of an exact shared prefix. The two stack: semantic caching skips the call entirely on a hit, prompt caching cheapens the calls you still make. In production you swap the toy embedder for a real model and the linear scan for an ANN vector index, and add a TTL plus LRU eviction so stale answers (prices, hours, policy) expire — and you never cache personalised or time-sensitive replies.

The takeaway I now carry: a huge share of real traffic is the same handful of questions reworded, so don't pay twice — cache the meaning, gate it on a threshold you tuned, and watch a big slice of the inference bill disappear.

Embed a question, watch it cosine-match the cache, and drag the threshold to trade hits against false hits:
https://dev48v.infy.uk/ai/days/day49-semantic-caching.html

Top comments (0)