Our production RAG pipeline wasn't slow because of embeddings.
It wasn't slow because of vector search.
It was slow because we kept answering the same question hundreds of times.
That's the whole story in three lines. Here's how we found it out, made it worse before we made it better, and landed on a fix that's honestly a bit boring — which is exactly why it works.
The problem: every query pays full price
Our original flow looked like this:
flowchart LR
U[User Query] --> E[Embedding]
E --> V[Vector DB Search]
V --> L[LLM Call]
L --> A[Answer]
Clean diagram. Expensive in practice. Every box costs time and money:
- Latency: 2–6 seconds round-trip per query, every time
- Cost: every LLM call burns tokens, even for a question we've already answered 500 times that day
- Repetition: users don't phrase things consistently
Here's the kind of thing we saw in our logs:
"What is the FD rate?"
"Fixed deposit interest?"
"Current FD interest?"
"FD rate for normal customers?"
Four sentences, one intent. Our pipeline had no idea. Each one triggered its own embedding call, its own vector search, and potentially its own LLM call — for a fact that hadn't changed since that morning.
Attempt #1: we fought an LLM problem with more LLM
Our first fix was to add an LLM in front of the pipeline as an intent-and-context detector, before the query ever touched the vector DB.
flowchart LR
U[User Query] --> I[LLM as Intent + Context Detector]
I --> E[Embedding]
E --> V[Vector DB]
V --> L[Main LLM Call]
L --> A[Answer]
It worked, in the sense that it correctly identified intent. But we'd added a full LLM call just to decide what to do next, and that decision itself cost us an extra 100–300 ms on every single query, before any real work even started.
We'd hired a security guard whose ID check took as long as the actual meeting.
| Approach | Latency added | Problem |
|---|---|---|
| No intent detection | 0 ms | Every query hits the full expensive pipeline |
| LLM-based intent detection | +100–300 ms | Accurate, but far too heavy for a routing decision |
One of our seniors summed it up in the retro: don't use a cannon to answer the door.
The actual fix: a small, boring classifier
We replaced the LLM-based detector with a small, fine-tuned classification model. No generation, no reasoning, no token-by-token output — just structured classification.
{
"intent": "RATE_QUERY",
"entity": "FIXED_DEPOSIT",
"confidence": 0.98
}
What we actually used:
- A distilled transformer encoder (DistilBERT/MiniLM class, roughly 30–60M parameters) — an encoder, not a generative decoder
- Fine-tuned on our own historical query logs, labeled by intent/entity pairs (
RATE_QUERY,BALANCE_QUERY,BRANCH_LOCATOR, etc.) - Exported to ONNX and quantized to INT8 for CPU inference — no GPU required for this step
- Deployed as a lightweight sidecar service, outside the main LLM call chain
The category shift mattered more than the specific model: classification and generation are different jobs. Asking "what is this question about" doesn't need a model that can write essays — it needs one that can sort mail. We'd been using a novelist to sort mail. Switching to an actual sorting machine took us from 100–300 ms down to single-digit milliseconds.
From intent to a Redis key
This is the part that usually gets glossed over when people explain RAG caching: we don't cache the user's raw sentence, because raw sentences are fragile. We build a canonical key from intent + entity instead.
"What is the FD rate?" → faq:rate:fixed_deposit
"Fixed deposit interest?" → faq:rate:fixed_deposit
"Current FD interest?" → faq:rate:fixed_deposit
"FD rate for normal customers?" → faq:rate:fixed_deposit
And here's what actually sits behind that key in Redis:
Key:
faq:rate:fixed_deposit
Value:
{
"answer": "The current FD interest rate for regular customers is 7.1% per annum.",
"updatedAt": "2026-07-20T09:15:00Z",
"ttl": 3600
}
Four different sentences collapse into one key. The first user to ask pays the full pipeline cost. Every user after that, for that key, gets served straight from Redis in milliseconds — regardless of how they phrased it.
One problem this immediately created
Caching answers is easy. Caching correct answers is the actual job.
One challenge we ran into almost immediately: FD rates change, but our cache didn't know that. A user could ask the rate on Monday, get it cached, and still be shown Monday's number on Thursday after the bank had already revised it. A confidently wrong cached answer is worse than a slow correct one.
We handled it with a short TTL (you can see it in the value above — 3600 seconds) plus an explicit invalidation hook that fires whenever the source rate data changes upstream. That's really its own topic, and I'll go deeper into cache invalidation strategy in the next article — but it's worth flagging here rather than pretending the caching layer was a clean win with no edge cases.
The full optimized flow
flowchart TD
U[User Query] --> C[Small Intent + Entity Classifier]
C --> K[Build Canonical Redis Key]
K --> R{Redis Key Cache}
R -- HIT --> ANS1[Return Cached Answer]
R -- MISS --> S{Semantic Cache}
S -- HIT --> ANS2[Return Cached Answer]
S -- MISS --> V[Vector DB Search]
V --> L[Main LLM Call]
L --> ANS3[Answer + Cache It]
Two cache layers, each catching a different case:
- Redis key cache — exact-match on canonical key, catches "same question, different words"
- Semantic cache — embedding similarity, catches "similar but not identical" queries before they hit the vector DB
Only genuinely new questions reach the expensive path: vector search + main LLM call.
How we actually validated this
Before rolling this out, we needed more than a hunch that it was faster. Here's the testing process:
- Replay testing — pulled 1 million real historical queries from production logs (anonymized) and replayed them through both the old and new pipelines side by side.
- Shadow mode — the classifier ran alongside the live pipeline without affecting real responses. We compared its predictions against actual downstream LLM behavior to catch misclassification early.
- Confidence threshold — we set a floor of 0.85. Anything below that didn't get trusted to hit cache; it fell through to the full pipeline. Slow-but-correct beats fast-but-wrong.
- Live A/B test — once shadow mode looked solid, we split real traffic and measured latency and cost directly in production, not just in a test harness.
Results across the 1M test cases:
| Metric | Before | After | Change |
|---|---|---|---|
| Avg latency | Full pipeline, every query | Cache hit on repeat intents | ~25–28% faster |
| LLM calls | 1 per query | Only on cache miss | ~25–28% fewer |
| Cloud/LLM cost | Full cost every query | Reduced by skipped calls | ~25–28% lower |
25–28% doesn't sound dramatic until you multiply it across a million queries — then it's a real line item on the cloud bill and a real difference in how fast users get answers.
Who built this
This wasn't a solo effort. The AI team owned the classifier — training data, labeling, model selection, threshold tuning. The backend team owned the plumbing — canonical key generation, Redis architecture, cache invalidation, and wiring it all together without breaking existing latency SLAs. Neither half works without the other.
What I'd tell someone building this next
- Don't fight an LLM's latency with another LLM. Match the tool to the job — classification isn't generation.
- A canonical key beats a clever prompt. It's what turns four different sentences into one cache hit.
- A cache without invalidation is a bug waiting to happen. Decide your staleness tolerance before you decide your TTL.
- Measure before you claim. A million replayed queries, shadow mode, and a confidence threshold are what make "25–28% faster" something you can defend on a whiteboard, not just a slide.
RAG isn't the hero in production. Caching is. And the intent classifier is what makes caching possible.
Next up: cache invalidation in depth — how we detect a rate change upstream and evict the right keys before Redis starts confidently lying to everyone.
Top comments (0)