Building a semantic cache layer in front of RAG — and why it might be the most underrated cost optimization in production AI systems.
Hey Dev community 👋
Every RAG architecture diagram I see looks exactly the same.
User → Vector Search → LLM → Response.
But I keep wondering...
Why are we paying for the same answer thousands of times?
Imagine 50,000 customers asking "How do I block my debit card?" in slightly different words. Why should the system perform 50,000 vector searches and 50,000 LLM calls to generate 50,000 nearly-identical answers?
Maybe I'm missing something. Here's the architecture that's been stuck in my head — and I want people who've run RAG in production to tell me where it falls apart.
Try this thought experiment before you keep reading: imagine you're building a banking chatbot with one million customers. If 70% of customer questions repeat every day, would you really pay for a vector search and an LLM call every single time? Or should your semantic cache eventually become the first place you look?
1. The Setup: A Bank's Customer Support Chatbot
Picture a bank's customer-facing chatbot that tries to resolve issues before a human agent gets involved.
Customer
│
▼
AI Chatbot
│
Answer?
│
Yes ──► Done ✅
│
No
│
▼
Human Agent
That chatbot does the same expensive thing under the hood for every question: vector search → LLM generation, every single time.
That's the piece I want to question.
Let's put numbers on it
Engineers care about numbers, not "thousands of users," so here's the scenario I'm imagining. These are illustrative assumptions, not measured data — I'm labeling them clearly so nobody mistakes this for a benchmark:
- 1,000,000 banking customers
- 120,000 support chats per day
- ~70% of those are FAQ-type, repetitive questions
- ~40 ms for a semantic cache lookup in Redis
- ~2–4 seconds for a full RAG pipeline call (vector search + LLM generation)
If 70% of 120,000 daily chats could be answered in 40ms instead of 2-4 seconds — and without a vector search or LLM call — that's not a minor latency win. That's a fundamentally different cost curve.
The sentence this whole article is built on
💡 Hypothesis
The more customers use the chatbot,
the less the chatbot depends on RAG.
To state it more precisely: my hypothesis is that RAG gradually shifts from being the default execution path to handling only cache misses and newly emerging questions. Every cacheable question enriches the semantic cache. Every repeated question increases the cache hit rate. Over time, the system becomes less dependent on expensive retrieval.
Everything below is me trying to work out whether that hypothesis actually survives contact with production.
2. The Standard RAG Flow (and its hidden tax)
Here's the textbook RAG pipeline most of us ship first:
User Question
│
▼
Embed Question
│
▼
Vector Search (Pinecone / pgvector / Redis)
│
▼
Retrieve Top-K Chunks
│
▼
LLM (generate answer using context)
│
▼
Response
This works. It's also stateless in the worst way — the system has zero memory that it already answered this exact question 40 times today. Every repeat question pays the full vector-search-plus-LLM tax again.
For a support bot fielding thousands of near-duplicate questions a day — "How do I block my debit card?", "My ATM card is lost", "Can I disable my card online?" — that's a lot of wasted spend on semantically identical work.
3. The Idea: A Semantic Cache in Front of RAG
Instead of a cache that only matches identical strings, what if we cache by meaning?
The first user pays the full RAG cost. Every similar question after that is an opportunity to avoid paying it again.
A semantic cache embeds the incoming question and compares it against previously answered questions using vector similarity. If the similarity score clears a threshold (say, 0.92 cosine similarity), it returns the cached answer — no vector search against the full knowledge base, no LLM call.
Why not just use a normal cache?
A normal Redis cache only works when two requests are exactly identical:
"How do I block my debit card?"
!=
"My ATM card is lost."
!=
"Disable my card."
Three different strings, three different cache keys, three cache misses — even though every one of them wants the same answer. A normal cache has no concept of meaning, only exact matches.
A semantic cache recognizes that all three express the same intent. That's why embeddings matter — they let the system compare questions by what they mean, not by how they're typed.
What the flow looks like
A normal cache would treat these as three different keys:
- "How do I block my debit card?"
- "My ATM card is lost. How can I disable it?"
- "I misplaced my card, what should I do?"
A semantic cache collapses all three into one lookup:
┌───────────────────────────┐
│ Semantic Cache │
│ (Redis + Embeddings) │
└─────────────┬───────────────┘
│
similarity ≥ threshold?
┌───────────┴───────────┐
YES NO
│ │
▼ ▼
Return Cached Answer RAG Pipeline
(milliseconds) │
Vector Search + LLM
│
▼
Store {embedding, answer} in Redis
│
▼
Return Response
The key shift: in this proposed architecture, RAG becomes the fallback path rather than the default.
4. What Should Actually Be Cached
This is where the idea breaks if you're not careful, so let's draw the line early.
Safe to cache — general knowledge-base questions with stable answers:
- "How do I block my debit card?"
- "What documents are needed for a personal loan?"
- "How do I reset my net banking password?"
- "What are the charges for an international transfer?"
Never cache — anything tied to live, personal, or account-specific state:
- Account balance
- Recent transactions
- Loan application status
- Credit card limits
Incoming Question
│
▼
Is this a policy/FAQ question, or does it need live account data?
│
┌────┴─────┐
POLICY/FAQ LIVE DATA
│ │
▼ ▼
Semantic Always hit live
Cache systems directly
eligible (never cached)
In practice, this usually means classifying intent before the cache lookup — a lightweight router that decides "cacheable knowledge query" vs. "personalized live query," and only sends the former through the semantic cache.
5. On Implementation
I'm deliberately not dropping 70 lines of Redis client code here — this article isn't about the API calls, it's about the architecture and the traffic pattern.
In practice, this could be built with Redis Vector Search, pgvector, or any other embedding store: embed the incoming question, run a KNN similarity lookup, return the cached answer above a threshold, otherwise fall through to the RAG pipeline and write the new answer back to the cache. The implementation isn't the interesting part — whether the traffic pattern I'm describing actually holds up is.
One thing I'd expect any real version of this to need: in production, I'd expect only validated or high-confidence answers to be written back into the semantic cache — not every RAG output. A hallucinated or low-confidence response getting cached and reused thousands of times would be worse than generating it fresh every time.
6. What If RAG Is Only a Bootstrapping Engine?
Maybe RAG isn't supposed to answer every question forever.
Maybe its real job is to answer new questions.
Once an answer proves useful and keeps getting requested — why shouldn't it graduate into the semantic cache?
Today
100 requests
│
▼
100 RAG calls
6 months later
100 requests
│
▼
80 Cache hits
20 RAG calls
If that's the right way to think about it, RAG's job quietly changes over time — from "answer everything" to "answer the unknown."
7. My Hypothesis (Not Production Data — Just a Guess)
I want to be upfront: I have no real measurements for this.
This is NOT production data. It's the mental model I'm trying to validate.
Here's the shape I'd expect if 80% of a support system's questions are repetitive and 20% are genuinely new:
Requests
100% |\
90% | \
80% | \
70% | \___
60% | \___ Expensive RAG requests
50% | \___
40% | \________
30% | ___________---- Semantic Cache hits
20% | ____/
10% |___/
0% |________________________________________ Time
Day 1 Week 1 Week 2 Month 1 Month 2 Month 3
| Time | RAG traffic | Cache hits |
|---|---|---|
| Day 1 | 100% | 0% |
| Week 1 | 85% | 15% |
| Week 2 | 70% | 30% |
| Month 1 | 55% | 45% |
| Month 2 | 40% | 60% |
| Month 3 | 25% | 75% |
If even half of this hypothesis holds, a mature semantic cache could become the primary retrieval layer for repetitive knowledge queries, with RAG handling only new, rare, or edge-case questions — cache invalidation, TTL expiry, and changing policies permitting.
8. Where I Think This Breaks (and I want you to tell me where else it does)
- Cache invalidation — when the knowledge base changes (a policy update, a new fee structure), how do you find and refresh every cached answer that's now stale? Do you version cached responses, or do you nuke the whole cache on any KB update?
- Similarity threshold tuning — set it too low and you return wrong answers for questions that only sound similar. Set it too high and the cache barely ever hits, killing the whole point.
- False positive risk — "How do I close my savings account?" and "How do I close my credit card?" are structurally similar sentences with very different answers. Embedding similarity alone might not be enough; some kind of entity/slot check might be needed on top.
- Memory management — Redis isn't free either. At what cache size does storage cost start eating into the savings from fewer LLM calls?
- Is Redis even the right tool, or would a dedicated vector cache (or a hybrid setup with a cheaper approximate index) make more sense at scale?
9. The Question I Can't Answer
Here's the question I can't answer:
If your cache hit rate eventually reaches 80%, is your semantic cache now your primary retrieval engine — and is RAG simply handling cache misses?
If that's a flawed way to think about production RAG, I'd genuinely love to know why.
Looking forward to the discussion
Top comments (2)
One thing I'm especially curious about: if you've built a production RAG system, what's your average semantic cache hit rate? Is it closer to 20%, 50%, or 80%? I'd love to compare this hypothesis with real-world experience.
"Maybe RAG's real job is to answer new questions" is a good line, and the architecture is sound. Two of your open challenges, cache invalidation and threshold tuning, actually share one fix: provenance. Store with every cached answer the IDs of the source chunks that produced it. Invalidation stops being a guessing game: when a policy doc changes, a reverse-index lookup tells you exactly which cached answers derived from it, and you evict those deterministically instead of waiting for a TTL or hunting stale entries by similarity.
On the threshold, the thing I'd push on is that the cost of errors is asymmetric. A cache miss costs you 2-4 seconds. A false hit serves a confidently wrong answer in 40ms with no model in the loop to hedge or refuse, which makes it the most expensive outcome in the whole system. So I would not tune 0.92 for hit rate; I'd tune it for false-positive cost, and back it with a cheap entity check before serving, since cosine similarity measures phrasing, not meaning. "Close account" vs "close credit card" will clear almost any threshold you pick, but an entity comparison catches it for near-zero latency. Similarity finds the candidate; something stricter should confirm it.