The short version
- Semantic caching stores your LLM's answers and returns a stored one when a new question means the same as an earlier one, so repeat questions never reach the model.
- An exact-match cache misses when a user drops a question mark or rewords the question. A semantic cache compares the questions' embeddings with cosine similarity, so paraphrases still hit.
- The right similarity threshold depends on the embedding model. With all-MiniLM-L6-v2 a reworded password question scored 0.89, so a 0.90 threshold would have missed it, while a different question with the same words scored 0.73.
- Put the semantic cache in front of prompt caching: a hit skips the LLM call, and a miss still gets the provider's cheaper rate for the cached prompt.
- Semantic caching pays off for support bots and FAQ agents that get many repeat questions. Skip it for coding agents, and never cache answers that depend on the user's account.
Semantic caching stores the answers your LLM gives and returns a stored answer when a new question means the same thing as an earlier one, without calling the model again. It compares questions by their embeddings rather than their exact wording, so "How do I reset my password?" and "I forgot my password, how do I reset it?" share one answer and one LLM call.
Below I go through how it works, how it compares to exact-match caching and prompt caching, and then a working semantic cache in Python that runs free on your own machine, first in memory and then in Postgres with pgvector. The finished code is on GitHub at hassanaziz0012/semantic-cache. The video above covers the same ideas.
The repeat question problem
Your users keep asking the same questions. "How do I reset my password?" "I forgot my password. How do I reset it?" "Help, I can't reset my password." All of these have the same answer: your chatbot tells the user where to go on your site to reset it.
But every time one of those duplicates comes in, the LLM processes all the tokens and writes the same answer from scratch, and you pay for every one of them. Some users add a question mark and some forget it, and to the model each version is a brand new request.
This mostly applies to support bots and FAQ agents, where users are bound to send lots of repeat questions. If you run agents like that, semantic caching can cut your token bill a lot.
How a semantic cache works
The cache sits between your users and your LLM. Take three people who forgot their passwords and ask your support agent about it.
The first question goes to the cache, and the cache has nothing yet. That's a cache miss, so the request goes on to the LLM, which writes an answer. The answer goes back to the user and gets stored in the cache next to the question.
The second and third questions also go to the cache first. This time the cache finds a stored question that means the same thing, which is a cache hit, and it returns the stored answer without calling the LLM.
So three LLM calls became one. Scale that up to a support bot that gets the same couple of hundred questions all day, and you skip hundreds of calls or more.
Exact-match caching vs semantic caching
A normal cache keys on the exact text. "How do I reset my password?" gets cached, and then a user asks "How do I reset my password" without the question mark. The text changed, so an exact-match cache misses.
A semantic cache keys on meaning. It scores how close two sentences are in meaning, so reworded and misspelled versions of a question can still hit.
Embeddings and similarity scores
To compare meaning, the cache turns text into vectors, long lists of numbers called embeddings. Words and sentences that mean similar things get similar numbers. "King" and "queen" end up close together, and "Mercury" ends up far from both, because it has nothing to do with either.
The same goes for whole questions. The three password questions all get embeddings that point in nearly the same direction, and comparing them gives a similarity score. The usual measure is cosine similarity, which is close to 1 when two embeddings point the same way. I explain vectors and cosine similarity in more detail in how semantic search works. Semantic caching is semantic search over your past questions.
What happens on each request
- A user sends a question to your application.
- Your application sends the question to an embedding model, which returns its embedding.
- The application searches a vector database for the closest stored question.
- If the closest one scores above your threshold, it's a hit: return its stored answer and skip the LLM.
- If not, it's a miss: call the LLM, send the answer to the user, and store the question's embedding with the answer for next time.
You need two pieces for this. The first is an embedding model. OpenAI and Gemini both have one, and Hugging Face has plenty of good open-source models you can run locally (the demo below uses one). The second is somewhere to store vectors and search them: pgvector (vectors inside Postgres), Pinecone, Chroma or any other vector database.
Semantic caching vs prompt caching
Prompt caching is a different trick that the LLM provider runs on their side. When the start of your prompt (usually the system prompt and any documents you include) is the same as a recent request's, the provider reuses the work it already did on that part and charges you less for those tokens. The model still writes a fresh answer every time. I cover it in how prompt caching works.
| Exact-match cache | Semantic cache | Prompt caching | |
|---|---|---|---|
| Matches on | The exact question text | The meaning of the question | An identical prompt prefix |
| On a hit | Returns a stored answer | Returns a stored answer | The model still answers, at a lower price |
| Calls the LLM on a hit | No | No | Yes |
| Who runs it | You | You | The LLM provider |
| Can return a wrong answer | Only if the answer went stale | Yes, if two different questions score as similar | No |
Use both. The semantic cache goes first: a hit never reaches the model. A miss goes to the LLM, where prompt caching makes the call cheaper because your long system prompt is already cached. Then the answer goes into the semantic cache so the next person asking gets it for free.
Build a semantic cache in Python
This version runs on your own machine with no API keys. It uses all-MiniLM-L6-v2, a small open-source embedding model from Hugging Face, and keeps the cache in memory. The LLM call is a stand-in function you swap for your real one.
python3 -m venv .venv
source .venv/bin/activate
pip install --index-url https://download.pytorch.org/whl/cpu torch
pip install sentence-transformers numpy
The first pip install gets the CPU-only build of PyTorch, which is a much smaller download than the default. Skip it if you have an NVIDIA GPU. Two files:
semantic-cache/
├── semantic_cache.py
└── app.py
semantic_cache.py
import time
import numpy as np
from sentence_transformers import SentenceTransformer
class SemanticCache:
def __init__(self, threshold=0.85, ttl_seconds=24 * 60 * 60):
self.model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
self.threshold = threshold
self.ttl_seconds = ttl_seconds
self.vectors = [] # one embedding per cached question
self.entries = [] # (question, answer, stored_at) in the same order
def embed(self, text):
# Normalized vectors make the dot product equal to cosine similarity.
return self.model.encode(text, normalize_embeddings=True)
def lookup(self, vector):
if not self.vectors:
return None, 0.0
scores = np.array(self.vectors) @ vector
now = time.time()
for i, (_, _, stored_at) in enumerate(self.entries):
if now - stored_at > self.ttl_seconds:
scores[i] = -1.0 # expired entries never match
best = int(np.argmax(scores))
score = float(scores[best])
if score >= self.threshold:
return self.entries[best][1], score
return None, score
def store(self, question, vector, answer):
self.vectors.append(vector)
self.entries.append((question, answer, time.time()))
embed turns a question into a 384-number vector. Because the vectors are normalized to length 1, multiplying the stored vectors by the new one gives the cosine similarity with every cached question in one step. lookup takes the best score and returns its answer only if it clears the threshold. Entries older than ttl_seconds (a day here) are skipped, so answers get regenerated after your docs change.
app.py
from semantic_cache import SemanticCache
llm_calls = 0
def ask_llm(question):
# Stand-in for your real model call (OpenAI, Anthropic, Gemini, a local model).
global llm_calls
llm_calls += 1
return f"LLM answer #{llm_calls}"
cache = SemanticCache(threshold=0.85)
def answer(question):
vector = cache.embed(question)
cached, score = cache.lookup(vector)
if cached is not None:
print(f"HIT {score:.2f} {question} -> {cached}")
return cached
reply = ask_llm(question)
cache.store(question, vector, reply)
print(f"MISS {score:.2f} {question} -> {reply}")
return reply
questions = [
"How do I reset my password?",
"I forgot my password. How do I reset it?",
"Help, I can't reset my password.",
"how can i change my password",
"How do I change my email address?",
"Can I update the email on my account?",
"My password reset email never arrived",
]
for question in questions:
answer(question)
print(f"{len(questions)} questions, {llm_calls} LLM calls")
answer is the whole flow from the list above: embed, look up, return on a hit, otherwise call the LLM and store the result.
Run it
python app.py
The first run downloads the model from Hugging Face. Then you should see this:
MISS 0.00 How do I reset my password? -> LLM answer #1
HIT 0.94 I forgot my password. How do I reset it? -> LLM answer #1
HIT 0.89 Help, I can't reset my password. -> LLM answer #1
HIT 0.86 how can i change my password -> LLM answer #1
MISS 0.44 How do I change my email address? -> LLM answer #2
MISS 0.60 Can I update the email on my account? -> LLM answer #3
MISS 0.73 My password reset email never arrived -> LLM answer #4
7 questions, 4 LLM calls
Four LLM calls instead of seven. All four password-reset questions share one answer, and "My password reset email never arrived" correctly gets its own, because it's a different problem that happens to use the same words. Each embedding took about 15 ms on my 4-core CPU, which is nothing next to an LLM call.
Picking the similarity threshold
The threshold decides everything. In the video I said 90 to 95 percent, and that's a fine starting point, but the right number depends on the embedding model. With this model, "Help, I can't reset my password." scores 0.89 against the first question, so a 0.90 threshold would miss it. I set 0.85 after looking at the scores.
Go too low and you get the dangerous mistake. "How do I reset my username?" scores 0.70 against the password question, and "Can I reset someone else's password?" scores 0.71. Neither should get the password answer. A miss costs you one LLM call, but a wrong hit gives a user the wrong answer with total confidence.
The output also shows a miss you'd want to catch: "Can I update the email on my account?" only scores 0.60 against "How do I change my email address?". Small models miss some paraphrases. That's the safe way to fail, and a bigger embedding model would catch more of them. Either way, log your questions with their best scores and pick the threshold from your real traffic.
Move the cache to Postgres with pgvector
An in-memory cache disappears when the process restarts. For a real app, store it in a vector database. Here's the same cache in Postgres with pgvector, run in Docker:
docker run -d --name semcache-pg -e POSTGRES_PASSWORD=test -p 5432:5432 pgvector/pgvector:0.8.1-pg18
pip install "psycopg[binary]" pgvector
Create the table with docker exec -i semcache-pg psql -U postgres < schema.sql:
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE semantic_cache (
id bigserial PRIMARY KEY,
question text NOT NULL,
answer text NOT NULL,
embedding vector(384) NOT NULL,
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX ON semantic_cache USING hnsw (embedding vector_cosine_ops);
Then pg_cache.py, with the same three methods as the in-memory version:
import psycopg
from pgvector.psycopg import register_vector
from sentence_transformers import SentenceTransformer
class PgSemanticCache:
def __init__(self, dsn, threshold=0.85):
self.model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
self.threshold = threshold
self.conn = psycopg.connect(dsn, autocommit=True)
register_vector(self.conn)
def embed(self, text):
return self.model.encode(text, normalize_embeddings=True)
def lookup(self, vector):
row = self.conn.execute(
"""
SELECT answer, 1 - (embedding <=> %(v)s) AS similarity
FROM semantic_cache
ORDER BY embedding <=> %(v)s
LIMIT 1
""",
{"v": vector},
).fetchone()
if row is None:
return None, 0.0
answer, score = row
if score >= self.threshold:
return answer, score
return None, score
def store(self, question, vector, answer):
self.conn.execute(
"INSERT INTO semantic_cache (question, answer, embedding) VALUES (%s, %s, %s)",
(question, answer, vector),
)
<=> is pgvector's cosine distance, so 1 - it gives the same similarity score as before, and the HNSW index keeps the search fast as the table grows. In app.py, change two lines:
from pg_cache import PgSemanticCache
cache = PgSemanticCache("postgresql://postgres:test@localhost:5432/postgres")
The first run prints the same output as before. Run it again and every question is a hit with 0 LLM calls, because the cache survived the restart. To expire old answers, run this on a schedule (a cron job is fine):
DELETE FROM semantic_cache WHERE created_at < now() - interval '1 day';
I delete old rows instead of filtering them in the lookup, because pgvector applies a WHERE filter after the index search, and a LIMIT 1 query can come back empty when the closest rows are the ones filtered out.
What not to cache
Only cache answers that are the same for everyone. "How do I reset my password?" is fine. "What's the status of my order?" is not, and a cache shared across users would show one customer's order to another. If answers depend on the account, skip the cache for those questions or keep a separate cache per customer.
Be careful with follow-ups too. "And how about my email?" means nothing without the earlier messages, so embed standalone questions, not every message in a conversation.
Where it pays off
Semantic caching works best when users send lots of repeat questions: support bots on your website, or a knowledge base assistant that answers from your documents. That's where you'll save the most.
If I were building a coding agent, semantic caching would be the last thing I'd add. Almost no two requests to a coding agent are the same, so the cache would almost never hit and you'd pay for an embedding on every request for nothing. If your app doesn't get many repeat questions, skip it.
If it does, the whole thing is as simple as it looks above: embed the question, look for a similar one in the cache, return its answer if you find one, and otherwise ask the LLM and cache what it says.
Originally published on hassandev.me.
Top comments (0)