Originally published at heycc.cn. This is a mirrored copy — the canonical version is kept up to date at the source.

Semantic Caching for LLM APIs: How Similarity-Based Response Caching Actually Works

A cache that matches on meaning instead of bytes sounds like a free lunch: paraphrase a question and still get an instant, free answer. The industry's own documentation tells a messier story. GPTCache's README states plainly that "you may encounter false positives during cache hits and false negatives during cache misses." Respan's engineering writeup publishes a threshold table showing that even a "balanced" 0.93 similarity threshold sends 3-7% of cache hits back with the wrong answer. TrueFoundry's engineering blog notes that "My left arm hurts" and "My right arm hurts" can score 0.91 similarity in a general-purpose embedding space — close enough to collide in a mistuned cache, in a domain where that collision matters.

Semantic caching is a real, measurable cost and latency win. It is also the only caching layer in the LLM stack that can silently hand a user the wrong answer with full confidence, because the failure mode isn't "cache miss, slower" — it's "cache hit, wrong." This piece covers the mechanics (embedding, nearest-neighbor lookup, threshold tuning), how it differs architecturally from prompt/context caching, what the savings actually look like with real numbers, and the mitigation patterns teams use in production to keep the correctness risk bounded.

The mechanism: embed, search, threshold, evaluate

A semantic cache sits in the application layer, between your app and the LLM API call. On every incoming query it runs a four-stage pipeline before deciding whether to call the model at all.

1. Embed the query. The incoming prompt (or, in multi-turn setups, just the final user message) is passed through an embedding model to produce a dense vector. GPTCache's architecture documents this as a pluggable stage — it supports OpenAI, Cohere, HuggingFace, ONNX, SentenceTransformers, fastText, and Timm embedding backends interchangeably, because the embedding choice materially affects what "similar" means downstream.

2. Nearest-neighbor lookup against a vector store. The query embedding is searched against previously cached query embeddings using an approximate-nearest-neighbor (ANN) index — GPTCache supports Milvus, FAISS, Hnswlib, PGVector, Chroma, and Qdrant as pluggable vector stores; Redis's LangCache and RedisVL's SemanticCache do the equivalent search directly in Redis. This returns the top-k closest cached entries and a distance (or similarity) score for each.

3. Apply a similarity threshold. This is the single knob that determines cache-hit aggressiveness, and every implementation exposes it slightly differently:

• GPTCache: similarity_threshold is a 0-1 config value (Config(similarity_threshold=0.8)). At threshold 0, every lookup is a hit; at threshold 1, nothing ever hits. The actual math for the FAISS backend: FAISS returns a raw L2 distance where smaller means more similar, with a backend-specific max_distance (4 for FAISS L2). GPTCache converts this to similarity_value = max_distance - search_distance, and a hit fires when similarity_value >= similarity_threshold * max_distance. At threshold 0.6 with max_distance=4, a hit requires similarity_value >= 2.4.
• Redis LangCache: default LANGCACHE_CACHE_THRESHOLD=0.65, tuned as an environment variable. Redis's own tutorial recommends starting at 0.65 for support-FAQ-style workloads, lowering it if obvious paraphrases are missed, and raising it if wrong cached answers surface for unrelated questions.
• LangChain's RedisSemanticCache: takes a score_threshold (default 0.2) — but this is a distance threshold, not a similarity score, so lower is stricter, the opposite direction from GPTCache's convention. This inversion is a real source of misconfiguration if you port threshold values between libraries without checking which convention each one uses.
• RedisVL's SemanticCache: distance_threshold on Redis COSINE distance scaled 0-2 (0 = identical, 2 = opposite), with a set_threshold() method for runtime tuning and set_ttl() for expiry.

4. (Optional) Run a second-pass similarity evaluator. Vector distance from an ANN index is a coarse, fast filter — it's not the only signal available. GPTCache separates "search" from "evaluate" as distinct pluggable stages and ships three evaluator strategies: ExactMatchEvaluation() (default, string equality), SearchDistanceEvaluation() (reuses the embedding distance), and EvaluationOnnx() (a cross-encoder ONNX model that re-scores the query/candidate pair with a more expensive, more accurate model). The point of splitting search from evaluation is that you can afford a cheap, high-recall ANN search over the whole cache, then a slower, high-precision re-ranker only on the handful of top-k candidates it returns — the same two-stage retrieve-then-rerank pattern used in production RAG pipelines.

Here's a concrete worked example from Redis's own LangCache tutorial. The cached query "How do I reset my password?" is compared against the paraphrase "I forgot how to change my login password." — different words, same intent. The call:

result = await lang_cache.search_async(
    prompt=question,
    similarity_threshold=self.similarity_threshold,
)

returns a match with similarity=0.833. Since the configured threshold is 0.65, this clears the bar and returns as a cache hit — no LLM call. That 0.833 score is also useful as a sanity check on your own threshold: if your production paraphrases are landing at 0.7-0.75, a threshold of 0.65 is barely permissive enough, and it's worth widening the margin before you tune down further.

Semantic caching vs. exact-match prompt caching: different layer, different economics

These two caching strategies are frequently confused because both are pitched as "LLM cost reduction," but they operate at different layers of the stack and save different things.

Dimension	Exact-match prompt caching	Semantic caching
Match condition	Byte-identical prompt prefix plus matching parameters (model, temperature, system prompt)	Embedding similarity of meaning, independent of exact wording
Where it lives	Inside the model provider's inference stack (Anthropic, OpenAI, etc. cache KV state server-side)	Application layer, between the user and the model API
What it saves on a hit	Input-token cost only (the cached prefix is billed at a discounted rate, not free)	Input and output token cost — the LLM call is skipped entirely
What breaks a hit	Any change to the prefix, including a single character, or a changed parameter	A query embedding that falls outside the similarity threshold
Correctness risk	Low — it's a literal string/state match, no semantic judgment involved	Real — "similar enough" is a judgment call the system makes on your behalf
Composability	Can run underneath semantic caching	Can run on top of prompt caching

The Redis engineering blog states the relationship directly: prompt/context caching "operates at the model-provider level," while semantic caching "lives in the application layer" — and because they solve different problems, "the two are complementary and can be stacked (double caching)." In practice this means: exact-match caching handles the case where you're re-sending the same long system prompt or few-shot examples on every call (cheap to implement, provider-managed, zero correctness risk), while semantic caching handles the case where different users ask the same underlying question in different words (bigger latency/cost win per hit, but requires you to own the correctness tradeoff).

The real cost and latency math

The vendor framing is aggressive — GPTCache's README headline claims "Slash Your LLM API Costs by 10x, Boost Speed by 100x." That's the project's own marketing claim with no published benchmark methodology attached to it in the README, so treat it as directional, not a number to build a cost model on.

The more defensible way to reason about savings is from first principles, because the mechanism tells you exactly where the savings come from:

• On a cache hit, you skip the LLM call entirely. Unlike prompt caching (which still bills a reduced input-token rate and still incurs the full generation latency for output tokens), a semantic cache hit returns a stored response directly from a vector/KV lookup. That means both the input-token cost and the output-token cost — typically the more expensive side of the bill for verbose completions — disappear on a hit.
• Latency drops from "network round-trip + model inference" to "embedding call + ANN search." An ANN lookup against an indexed vector store is milliseconds; embedding a short query is also low-latency, especially with a local/ONNX embedding model instead of a remote embedding API call. The end-to-end latency on a hit is dominated by whichever of those two steps is slower, not by token-by-token generation.
• The savings are proportional to your hit rate, and hit rate is workload-dependent. A high-repetition workload — customer support FAQs, onboarding flows, common coding questions — will see meaningfully higher hit rates than a workload with mostly unique, long-tail queries (open-ended research assistants, creative writing). This is why GPTCache's own docs recommend monitoring Hit Ratio, Latency, and Recall as the three operating metrics, rather than assuming a fixed savings percentage applies to your traffic.

The practical framing: semantic caching is a multiplier on your duplicate-intent traffic, not a flat discount on all traffic. Before adopting it, it's worth instrumenting how much of your actual query volume is semantically repeated — if that number is low, the infrastructure and correctness-risk overhead may not pay for itself.

The correctness risk: what "similar enough" actually means in practice

This is the part the marketing pages gloss over. A vector-similarity score is a proxy for semantic closeness in whatever the embedding model learned — it is not a proxy for "these two queries have the same correct answer." Those two things overlap a lot, but not always, and the gap between them is exactly where wrong cache hits live.

The clearest artifact for this is Respan's own threshold/hit-rate/false-positive table, published in its semantic caching writeup. It's worth reproducing in full because it's the closest thing in this space to a documented precision/recall tradeoff curve for cache thresholds:

Similarity threshold	Hit rate	False-positive rate
0.99	1-3%	under 0.1%
0.97	5-10%	~0.5%
0.95	15-25%	1-3%
0.93	25-40%	3-7%
0.90	35-55%	7-15%
0.85	45-70%	15-30%

Read this table as a single tradeoff curve, not six independent data points: hit rate roughly triples going from 0.99 to 0.90, and false-positive rate roughly grows in step. There is no threshold on this curve where hit rate is high and false-positive rate is negligible — every gain in cost savings (higher hit rate) is bought with a specific, quantified increase in the chance of serving a wrong answer. That tradeoff, not any single "correct" threshold, is the actual finding.

Respan pairs this table with an explicit tolerance guideline rather than a single universal number: it recommends capping the acceptable false-positive rate at 2% for non-regulated systems and 0.5% for regulated ones. Read against the table above, that guidance rules out the 0.93 "balanced" threshold (3-7% FP) for both categories — it only clears 0.5% at 0.97-0.99, and only clears 2% somewhere between 0.95 and 0.97. In other words, Respan's own numbers argue for thresholds in the high-0.9s for anything where a wrong answer has real cost, not for treating a mid-single-digit false-positive rate as an acceptable steady state.

Documented false-positive collisions:

• Respan's engineering writeup gives a code-generation example: "Sort an array" and "sort an array in descending order" embed at approximately 0.94 cosine similarity — high enough to clear almost any reasonable threshold in the table above — yet the correct answers differ materially (ascending vs. descending sort logic). High similarity does not guarantee interchangeable answers for instruction-following or code queries, where a single qualifier flips the correct output. It also illustrates why the false-positive rates in Respan's table are non-zero even at aggressive-looking thresholds like 0.93: some wrong-answer pairs simply embed closer together than many correct-answer paraphrases do.
• TrueFoundry's engineering blog gives a general-embedding-space example: "My left arm hurts" and "My right arm hurts" can score around 0.91 similarity. At a 0.90 threshold, a cache would return identical advice for two queries describing opposite-side symptoms — cosmetically minor in a coding assistant, and a real problem in any domain where left/right, positive/negative, or included/excluded distinctions change the correct response. TrueFoundry's own framing for the risk band this falls into is blunter than the score alone conveys: "For a general FAQ bot 0.88 may be perfectly safe. For triage it is malpractice" — the point being that the same threshold is safe or reckless entirely depending on what's riding on a wrong answer, not on the score itself.

The other failure direction: staleness. A cached response can be similarity-correct and still be factually wrong if the underlying data changed after it was cached — pricing pages, API rate limits, weather, anything time-sensitive or backed by a document that got edited. GPTCache's docs explicitly call this out, instructing developers to skip the cache (or use exact-match-only / a very short TTL) for queries like "What's the weather in..." or anywhere the knowledge base has changed since the entry was cached.

A secondary, easy-to-miss risk: eviction is naive. GPTCache currently evicts purely by cache-line count (LRU/FIFO/LFU/RR policies keyed on entry count), not by memory footprint — the docs flag this as an open limitation. If your cached responses vary wildly in size, count-based eviction can misjudge actual memory pressure, either evicting too aggressively or letting memory grow further than expected.

Mitigation patterns used in production

None of these risks are arguments against semantic caching — they're arguments for treating threshold selection and cache scope as a tuned system, not a config default you ship and forget.

1. Multi-factor matching, not similarity alone. TrueFoundry's pattern: only the final user message goes through semantic comparison. System prompt, model name, temperature, conversation history, and tenant/user ID are required to match exactly before a cache hit is even considered. This alone eliminates an entire class of false positives — a semantically similar question asked under a different system prompt or by a different tenant should never hit the same cache entry, regardless of embedding distance.

2. Entity/keyword guards on top of vector similarity. Before serving a hit, check that salient entities — country, order number, product name, left/right, error code — match between the query and the cached entry. Cosine similarity has no built-in mechanism to enforce entity agreement; "my left arm hurts" and "my right arm hurts" is exactly the case this guard catches that pure vector search misses.

3. Domain-tiered thresholds, anchored to a sane default. TrueFoundry's own starting point is a 0.9 threshold, adjusted from there by domain: 0.95-1.0 is "strict — only nearly identical queries match," recommended for "high-precision scenarios where incorrect cache hits carry significant cost"; 0.85-0.95 is "balanced — works well for most conversational apps," aimed at general-purpose chatbots and FAQ systems; below 0.85 is "broad — may return loosely related answers," reserved for exploratory or low-risk workloads. The threshold isn't a universal constant — it should be set per use case based on the cost of a wrong answer in that domain, starting from 0.9 and moving in the direction the false-positive table above implies.

4. Shadow-mode testing before going live. Run the semantic cache in observation-only mode against real production traffic for at least a week, logging what would have matched at each candidate threshold without actually serving cached responses. Plot precision/recall curves specific to your domain and traffic before flipping the cache on for real users — this converts threshold selection from a guess into a measured decision, and it's the only reliable way to find out whether your own traffic's false-positive rate at a given threshold tracks Respan's table or looks different because your query distribution is narrower or wider.

5. Exact-match-only or short-TTL handling for time-sensitive content. For prices, rates, availability, or anything else that changes independent of the query wording, either bypass semantic matching entirely (require exact match) or attach an aggressive TTL so stale entries expire quickly regardless of how "similar enough" future queries look.

6. Content-hash invalidation for RAG-backed caches. When cached responses are grounded in retrieved documents, include a content hash or version ID of the source document in the cache entry's metadata. When the source document changes, its hash changes too, so any cache entry built on the old version naturally misses on the next lookup — closing the staleness gap without needing to manually track which cache entries depend on which documents.

7. Monitor the three operating metrics continuously, not just at launch. GPTCache's own guidance — Hit Ratio, Latency, and Recall — is a minimum bar. A cache that looks healthy at rollout can drift as traffic patterns shift (new features, new user segments, seasonal query changes), so these need to be dashboarded, not checked once.

When semantic caching is (and isn't) worth the correctness tradeoff

Signal	Lean toward semantic caching	Lean toward skipping it (or exact-match only)
Query repetition	High — FAQ-style, support bot, onboarding flows with predictable phrasing variance	Low — mostly unique, long-tail, or creative/open-ended queries
Cost of a wrong answer	Low-to-moderate — a slightly-off code suggestion or FAQ answer is correctable	High — medical, legal, financial, or safety-relevant guidance where a wrong answer causes real harm
Data freshness	Static or slow-changing knowledge (product docs, general how-tos)	Time-sensitive (prices, availability, weather, live status)
Query structure	Coarse-grained intent questions where paraphrase = same answer	Fine-grained instructions where a single qualifier changes the correct output (e.g., "ascending" vs. "descending", "left" vs. "right")
Multi-tenancy	Single-tenant or cache correctly scoped per tenant/session	Multi-tenant without strict tenant-ID exact-match guards
Team capacity	Willing to instrument hit ratio/latency/recall and run shadow-mode tuning	Want a set-and-forget cache with no ongoing tuning budget
Acceptable FP rate (per Respan's table)	Workload tolerates 1-3% at threshold 0.95, or up to ~7% at 0.93	Workload needs Respan's regulated-domain ceiling (0.5%), which requires threshold 0.97+ and a correspondingly lower hit rate

Worked configuration example (GPTCache)

To make the threshold mechanics concrete, here's the minimum GPTCache setup that wires similarity search to a threshold-gated evaluator, following the library's documented Config pattern — actually installed and run against a local SentenceTransformers model (all-MiniLM-L6-v2), not left as an untested snippet:

from gptcache import cache
from gptcache.manager import get_data_manager, CacheBase, VectorBase
from gptcache.similarity_evaluation.distance import SearchDistanceEvaluation
from gptcache.embedding import SBERT
from gptcache.processor.pre import get_prompt
from gptcache import Config
from gptcache.adapter.api import get, put

embedding_func = SBERT("all-MiniLM-L6-v2")

# Storage: SQLite for responses, FAISS for embeddings
data_manager = get_data_manager(
    CacheBase("sqlite"),
    VectorBase("faiss", dimension=embedding_func.dimension)
)

cache.init(
    pre_embedding_func=get_prompt,   # plain-string mode, not chat-message mode
    embedding_func=embedding_func.to_embeddings,
    data_manager=data_manager,
    similarity_evaluation=SearchDistanceEvaluation(),
    config=Config(similarity_threshold=0.8),
)

put("What is the capital of France?", "The capital of France is Paris.")

One real gotcha surfaced by actually running this rather than trusting the documented Config pattern in isolation: cache.init() without an explicit pre_embedding_func defaults to a chat-completions pre-processor that expects {"messages": [...]} and raises TypeError: 'NoneType' object is not subscriptable on a plain string prompt. pre_embedding_func=get_prompt (from gptcache.processor.pre) is required for the plain-string get()/put() API shown here — a detail the library's own Config-focused examples don't make obvious until you actually hit the error.

With that fixed, real output from issuing two follow-up queries against the cached entry above:

Paraphrased query ("What's the capital city of France?"):
  raw distance = 0.0866, cache result = 'The capital of France is Paris.'  (HIT)

Unrelated query ("What is the population of Japan?"):
  raw distance = 1.5494, cache result = None  (MISS)

The gap between 0.0866 and 1.5494 is the whole mechanism made concrete: SearchDistanceEvaluation is comparing raw vector distance, and the paraphrase lands close enough to clear a 0.8 similarity threshold while the unrelated query doesn't come close.

With similarity_threshold=0.8 and the FAISS backend's max_distance=4, a hit requires similarity_value >= 3.2 (0.8 * 4) — a tight bar that favors precision over recall. Cross-referencing against Respan's table (which uses cosine similarity, not FAISS's L2-derived similarity_value, so the numbers aren't directly interchangeable but the shape of the tradeoff transfers): a threshold this conservative sits above even the 0.93 "balanced" band, which is the right region to start from given that band's documented 3-7% false-positive rate. Loosen it deliberately, in shadow mode, rather than starting loose and discovering the collision rate in production.

Where this fits with the rest of the LLM cost stack

Semantic caching is one lever among several for controlling LLM API spend — it composes with, rather than replaces, prompt caching, batch APIs, and model routing (covered in depth in How to Reduce LLM API Costs in 2026). The embedding model you choose for the cache's similarity search is the same category of decision covered in How to Choose an Embedding Model in 2026 — and the vector store backing the cache (FAISS, Redis, Qdrant, etc.) draws from the same tradeoffs laid out in Vector Database Comparison 2026, since a semantic cache is, mechanically, a small specialized RAG system where the "documents" are past queries and the "answers" are past responses.

Where these numbers came from (checked 2026-07-02)

Checked directly against primary sources, with the Respan and TrueFoundry citations re-verified word-for-word after an editorial pass flagged discrepancies in an earlier draft:

• GPTCache: similarity_threshold semantics, the Config class default, and the FAISS max_distance/similarity_value formula checked against GPTCache's own docs/usage.md, GitHub Discussions thread #577, and the project README (architecture, embedding/vector-store backends, evaluator classes, eviction policies, and the "10x cost / 100x speed" headline claim, which is flagged above as an unverified vendor claim rather than an independently benchmarked figure).
• Redis LangCache: default threshold (LANGCACHE_CACHE_THRESHOLD=0.65) and the worked 0.833-similarity password-reset example checked against Redis's own LangCache tutorial (redis.io/tutorials/semantic-caching-with-redis-langcache/), including the exact code snippet and query pair.
• Respan: the threshold/hit-rate/false-positive-rate table (0.99 through 0.85), the 2%-non-regulated/0.5%-regulated tolerance guidance, and the "Sort an array" vs. "sort an array in descending order" ~0.94-similarity example were re-checked verbatim against respan.ai/articles/semantic-cache-llm. An earlier draft of this article attributed a "production writeup...naive 0.85 threshold" anecdote and a "similarity-score distributions overlap in the 0.85-0.92 band" claim to Respan, and separately described a 3-5% false-positive rate as Respan's recommended "practical ceiling" — none of that is present in the source, and the last point inverted Respan's actual guidance (which treats 3-7% at threshold 0.93 as too high, not as an acceptable target). All three have been removed and replaced with the source's real table and tolerance guidance above.
• TrueFoundry: the "My left arm hurts" / "My right arm hurts" ~0.91-similarity example, the three threshold bands (0.95-1.0, 0.85-0.95, below 0.85) with their exact "strict / balanced / broad" descriptions, the 0.9 starting-point recommendation, and the multi-factor exact-match pattern (system prompt, model, temperature, history, tenant/user ID) were re-checked verbatim against TrueFoundry's blog post (truefoundry.com/blog/semantic-caching-llm-gateway). An earlier draft mischaracterized this as a "medical-triage-chatbot" case study and mislabeled the bands as being organized around "medical" as a named domain — the source's own framing is risk/precision-tolerance based, not domain-named, and that language has been corrected above.
• LangChain RedisSemanticCache: the score_threshold default of 0.2 as a distance (not similarity) threshold is consistent with publicly documented LangChain/langchain-redis defaults; note that some newer versions rename the parameter to distance_threshold.
• The worked GPTCache configuration example was actually installed and run (pip install gptcache sentence-transformers) rather than left as an untested snippet — surfacing a real gotcha (the default chat-completions pre_embedding_func errors on a plain-string prompt; get_prompt is required) and producing genuine raw-distance output for a paraphrase (0.0866, HIT) versus an unrelated query (1.5494, MISS) against a local all-MiniLM-L6-v2 embedding model.

Sources

• GPTCache GitHub repository and README
• GPTCache usage documentation
• GPTCache GitHub Discussion #577 — similarity threshold and distance formula
• Redis LangCache semantic caching tutorial
• Redis engineering blog — prompt caching vs. semantic caching
• Respan — Semantic caching for LLM applications
• TrueFoundry — Semantic caching for LLM gateways