DEV Community

Cover image for Would you serve a cached LLM answer to a question that only looks the same? (Friday AI question)
M.T.Ramkrushna
M.T.Ramkrushna

Posted on

Would you serve a cached LLM answer to a question that only looks the same? (Friday AI question)

Friday AI Question. This one is about the cheapest LLM call: the one you never make.

A user asks your support bot a question that is almost the same as one it answered an hour ago. Do you call the model again, or serve the cached answer?

Pick one (and tell me in the comments):

  • A) No cache. Every request goes to the model.
  • B) Exact-match cache only. Same normalized prompt, same answer.
  • C) Semantic cache with one similarity threshold for everything.
  • D) Exact-match first, then a semantic cache only for scoped, non-personal questions, with a TTL and a threshold you tuned by measuring wrong hits.

My lean: B everywhere, D where you have lots of repeat questions. Here's why.

What a semantic cache does

An exact-match cache only helps when the prompt is byte-for-byte the same. Real users rarely type the same thing twice. A semantic cache embeds each incoming question, finds the nearest cached question in a vector store, and if the two are "close enough", returns the stored answer without calling the LLM.

The savings can be real. In GPT Semantic Cache (Regmi & Pun, 2024), a Redis-backed semantic cache reduced API calls by up to 68.8% across their query categories, with cache hit rates from 61.6% to 68.8%, and they report positive hit rates (cached answers judged correct) above 97%.

Note what those categories were: customer-support style questions such as technical support, order/shipping and shopping Q&A. That is exactly the kind of traffic where the same question really does come in many times.

The catch: "close enough" is a single number

Every semantic cache has a knob that decides what "close enough" means:

  • RedisVL SemanticCache uses distance_threshold, measured in Redis cosine distance (0 to 2, lower is stricter), and it defaults to 0.1. It also takes a ttl and filterable_fields.
  • GPTCache (Zilliz) uses a similarity_threshold from 0 to 1 that defaults to 0.8, plus pluggable similarity evaluators, from a simple distance check to model-based and rerank evaluators.

The problem with one static number: two questions can be very close in embedding space and still need different answers. Think "How do I cancel my order?" versus "How do I cancel my subscription?" (my example, not from a paper). A threshold loose enough to catch real paraphrases may also catch these.

This isn't just a hunch. The vCache paper (Schroeder et al., Verified Semantic Prompt Caching, ICLR 2026) observes that static thresholds used across all requests give no formal correctness guarantees, lead to unexpected error rates, and give suboptimal hit rates. Their fix learns a separate threshold per cached prompt online, against a user-set maximum error rate, and they report up to 12.5x higher cache hit rates and 26x lower error rates than static-threshold and fine-tuned embedding baselines.

A wrong cache hit is worse than a slow answer: it's a confident answer to a question the user didn't ask, and it looks exactly like a normal response.

Why I land on B, then D

A (no cache) is fine at low volume, but you pay full price and full latency for every repeat.

B (exact-match) is almost free and can't return an answer to a different question. Normalize whitespace and case if you like, but keep it exact.

C (one global threshold) is where the wrong hits come from: personal questions, questions about different entities that read alike, and answers that went stale.

D keeps the savings and fences the risk:

  1. Scope the cache. Only cache non-personal, non-account-specific questions. Partition entries with filters (tenant, language, product, prompt/model version) so a hit can only come from the same context. RedisVL's filterable_fields exist for this.
  2. Expire entries. Set a TTL so answers about prices, policies or docs don't outlive the source.
  3. Tune the threshold on your own data. Collect pairs of real paraphrases and "look-alike but different" questions, and pick the threshold by measuring wrong hits, not by keeping the default.
  4. Log every hit with the matched question and its distance, so you can audit and evict bad entries.
  5. Watch the right metrics: hit rate, wrong-hit rate (sampled and reviewed), and cost and latency saved.

When not to use a semantic cache

  • Answers that depend on the user's own data or account state.
  • Fast-changing facts.
  • Low-repeat traffic, where you pay for an embedding lookup on every request and rarely hit.

References

So: A, B, C or D? And if you run a semantic cache today, how did you pick the threshold?

Top comments (0)