A semantic cache hit is a claim. It says: "this new prompt deserves exactly the answer I gave to an older one." Most of the time that claim is true, and you save a model call. When it's false, the user gets a confident, fast, wrong answer, and your dashboard logs it as a win.
So instead of asking "how high should my similarity threshold be?", I find it more useful to ask a different question: which prompts should never be answered from memory, no matter how similar they look?
Below are seven of them, each a real example from a semantic cache I built for Microsoft.Extensions.AI 10.11 on .NET 10. It's a DelegatingChatClient that embeds the prompt, searches a Microsoft.Extensions.VectorData collection, and serves a stored answer only when that's safe. It runs fully offline with a fake chat model and a fake embedder, and it has 97 tests (85 unit, 12 integration). The scores quoted below come from that offline embedder, which deliberately copies the weaknesses of real ones. Code is trimmed to signatures and shape; the omitted parts are marked.
Tech Skill Builder is $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10. The complete semantic cache project from this post, with a step-by-step PDF, is inside Tech Skill Builder.
Two places a prompt gets refused
It helps to know there are two separate gates, because the seven prompts split between them:
- Before the lookup. An eligibility check decides whether this call may touch the cache at all. If not, it goes straight to the model and nothing is stored. This is the cheapest refusal: no embedding, no vector search.
- After the lookup. The vector search finds a close candidate, and a cheap deterministic guard decides whether that candidate is actually the same question.
public CacheDecision Evaluate(IEnumerable<ChatMessage> messages, ChatOptions? chatOptions,
string? tenantId, bool forExactCache = false)
{
// opt-out flag, no tenant, conversation state, tools, temperature,
// tool-call history, non-text content, multi-turn, prompt length, personal data
// (full rule set in the complete project)
}
public static GuardVerdict Compare(PromptSignature probe, PromptSignature cached)
{
// negation, entity/id, opposite-action checks (omitted here)
}
Every refusal is recorded with a reason (bypass with the rule that fired, or guard_reject), so you can see them on a dashboard.
Prompt 1: "Do not cancel my order 1234"
Cached earlier: "Cancel my order 1234". Similarity: 0.985. That clears a 0.92 threshold with room to spare.
Embeddings capture what a sentence is about, and both of these are about cancelling order 1234. The word "not" barely moves the vector. Replaying the cached answer tells the user their order is being cancelled when they asked for the opposite.
Refused by: the guard. Each stored entry keeps a small signature of its prompt, including "is it negated?". If the new prompt and the candidate disagree, the candidate is rejected and the model is called. Contractions like "don't" are expanded first, so they count too.
Prompt 2: "Cancel my order 5678"
Cached: "Cancel my order 1234". Similarity: 0.970.
Same intent, different order. A number is a tiny part of the sentence, so it's a tiny part of the vector. But for the user it's the whole point.
Refused by: the guard's entity check. The signature lists the numbers, order-style ids (like ORD-1234) and quoted strings in the prompt. If the sets differ, no hit. "Cancel my order 12345" is rejected for the same reason.
Prompt 3: "How do I disable two-factor authentication?"
Cached: "How do I enable two-factor authentication?". Similarity: 0.980.
Opposite verbs in the same context embed almost identically. Serving the "enable" steps to someone trying to disable it is harmless-looking and very confusing.
Refused by: the guard's opposite-action check, a short list of pairs like enable/disable, add/remove, subscribe/unsubscribe, lock/unlock. If both prompts use the same pair but different sides, it's a reject.
Is that list crude? Yes. It's English-only regex and word lists. That's fine for its job: it doesn't need to understand language, only to veto the specific mistakes vector similarity is known to make. With a real embedder you'd grow the list from your own traffic.
Prompt 4: "How do I reset my password?" (from another tenant)
Tenant acme asked this an hour ago and the answer is cached. Now tenant globex asks the identical question. Similarity doesn't matter here, because their answers can differ: different product, different reset flow, maybe different policy.
Refused by: the lookup itself, before similarity is even considered. Each entry belongs to a partition, a hash of everything that shapes the answer other than the user's words: tenant, model, system prompt, the answer-shaping options (temperature, max tokens and so on), a version tag, and the embedding model. The vector search only looks inside one partition. And a call with no tenant at all skips the cache entirely. Failing closed is cheaper than explaining a cross-tenant leak.
Prompt 5: "Where is my order right now?" (with tools enabled)
If the call offers the model a track_order tool, the real answer depends on what that tool returns right now. A cached answer from ten minutes ago is a snapshot of old data dressed up as a fresh reply.
Refused by: eligibility. Any call with tools in ChatOptions, or with function calls or results in its history, bypasses the cache. If tool data is expensive, cache the tool's data with its own expiry, never the model's summary of it.
Prompt 6: "And for gift cards?"
On its own this means nothing. Its meaning lives in the previous turn ("What's your refund policy?"). Embed only the last message and you might match some unrelated cached question about gift cards.
Refused by: eligibility. The semantic layer is single-turn: if the conversation already contains an assistant message, or more than one user turn, it bypasses. Calls carrying server-side conversation state (a conversation id or continuation token) bypass too, because that state isn't visible to the cache.
The exact-match cache that sits in front of it is allowed to handle multi-turn, because its key hashes the whole history.
Prompt 7: "My email is mona@example.com, why was I charged twice?"
Two problems. The answer is about one specific person, so it should never be served to anyone else. And storing the prompt means personal data now sits in a shared vector index.
Refused by: eligibility, through a deliberately blunt detector: emails, card-like digit runs, phone numbers, and phrases like "my passport" or "my IBAN". A false positive costs one model call. A false negative can leak someone's details. In production I'd swap the regexes for a proper PII classifier.
Honourable mentions
- High temperature. If the caller asked for variety (above 0.3 in the sample), giving them a stored answer quietly ignores the request.
-
"How long does standard shipping take to Canada?" It scored 0.894 against the domestic shipping question, below 0.92, so it was correctly refused. It shows up as a
near_miss(within 0.07 below the threshold), which is the band to watch when someone wants to lower the threshold.
How do you know the refusals work?
Treat the cache like a classifier, where "serve the cached answer" is a positive prediction. The project has a small golden set of seed prompts and labelled probes: paraphrases that should hit, plus negations, id swaps, opposite actions, near-topic and cross-tenant prompts that must not. With the offline embedder at 0.92:
- threshold only: precision 0.688, five false hits (all of them negations, id swaps or opposite actions)
- threshold plus guard: precision 1.000, recall unchanged at 0.917
A unit test fails the build if guarded precision drops below 1.0. Those numbers belong to the toy embedder. Point the same evaluation at a real embedding model and they will change, which is exactly why the golden set should run in CI.
The short version
- [ ] Negated prompts never reuse non-negated answers
- [ ] Different ids or numbers never share an answer
- [ ] Opposite actions are vetoed after the threshold
- [ ] The tenant (and model, system prompt, options, version) is part of the partition
- [ ] Tool calls and tool history bypass the cache
- [ ] Follow-up turns bypass the semantic layer
- [ ] Anything that looks personal is neither looked up nor stored
- [ ] A golden set with hard negatives gates the build
What I didn't verify
Everything above ran against the offline fake embedder and chat model. The project can switch to OpenAI with a key, but I haven't measured these scores or the guard against a real embedding model, and the default 0.92 threshold was chosen for the fake. Recalibrate before trusting it.
Get the complete project
The snippets here are the outline. The complete, working .NET 10 solution is in Tech Skill Builder: the SemanticCachingChatClient middleware (including streaming), a tenant-aware exact cache in front of it, the scope builder, the full eligibility rules and prompt guard, a VectorData store you can swap for Redis, Qdrant or Postgres, telemetry, the golden set and evaluator, per-tenant threshold calibration, an ASP.NET Core minimal API, a CLI demo and all 97 tests, plus an in-depth PDF guide. No API key needed to run it.
Tech Skill Builder gives you complete, working, tested .NET projects with full source code and step-by-step PDF guides, with new ones added regularly. Tech Skill Builder is $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10. Join Tech Skill Builder
Which prompt has your cache answered from memory that it really shouldn't have?
Top comments (0)