I Built a Semantic Cache for RAG. The Hard Part Was Knowing When NOT to Cache.

Yatin Annam — Fri, 09 Oct 2026 15:46:52 +0000

Every time a RAG application answers a question, it may need to retrieve documents and make an LLM call.

But what happens when someone asks essentially the same question again?

We could reuse the previous answer. The challenge is knowing when two questions are similar enough to share an answer—and when they only look similar.

That's the problem I explored while building Weir, an open-source, cost-aware gateway for Retrieval-Augmented Generation (RAG).

GitHub: https://github.com/yatinannam/weir

What is Weir?

Weir sits in front of an existing RAG service. It decides whether a request can safely use a cached answer or needs retrieval and generation, and chooses a model tier when generation is necessary.

It combines three main ideas:

Semantic caching: Reuse answers for sufficiently similar questions.
Model routing: Use a cheaper model for suitable questions and a larger model when needed.
Cost accounting: Track actual request cost, counterfactual cost, and latency so savings can be measured.

It also includes request logging, monitoring, evaluation tooling, and a Docker-based demo.

The surprising problem with semantic caching

Suppose a user asks:

"What are the ICU visiting hours?"

Later, another user asks:

"What are the general ward visiting hours?"

The questions are structurally similar. An embedding-based cache might assign them a high similarity score.

But serving the ICU answer to the second user would be incorrect.

This is why Weir uses an entity guard alongside embedding similarity. It checks distinguishing information such as numbers, negation, and important domain terms before accepting a potential cache hit.

The goal isn't to cache as aggressively as possible. It's to save unnecessary work without silently returning the wrong answer.

How I evaluated it

I compared four configurations:

A baseline that always uses the larger model without caching
Semantic caching alone
Model routing alone
Full Weir, combining caching and routing

The evaluation used a fictional hospital FAQ with 40 documents and 158 evaluation questions, including paraphrases, look-alike questions, and unanswerable questions.

1. Cold-pass evaluation

Each of the 158 questions was asked once.

Full Weir reduced measured cost per 1,000 requests from $0.0988 to $0.0786—a 20% reduction. The evaluation judge score and key-fact score remained unchanged against the baseline.

The trade-off matters: median latency increased from 767 ms to 876 ms in this workload. Cost reduction does not automatically mean every workload becomes faster.

2. Repeat-heavy replay

I also tested a 300-request replay in which 73.7% of requests were repeats.

Metric	Baseline	Full Weir
Cache hit rate	0%	93.0%
Cost per 1,000 requests	$0.0970	$0.0055
Median latency	747 ms	14 ms
Judge score	4.79	4.79
Key-fact score	0.947	0.948

On this workload, the measured cost reduction was approximately 94%, and median latency was about 50 times lower.

The important qualification is that these are synthetic, repeat-heavy benchmark results—not proof of equivalent savings on a production workload.

What about incorrect cache hits?

At the chosen similarity threshold, embedding similarity alone accepted some dangerous look-alike matches in the evaluation set. Adding the entity guard reduced wrong accepted cache hits to zero on that test set.

That is encouraging, but it is not a guarantee of safety. A small synthetic dataset cannot cover every possible ambiguity or domain-specific distinction.

What happened under load?

I also tested the system under load, including provider failures and database delays.

The tests surfaced a genuine overload mode: when cache lookups began timing out under CPU pressure, more requests bypassed the cache and reached retrieval and generation, adding further load.

I documented that limitation rather than presenting the load test as an unconditional capacity claim.

Try it yourself

Weir includes a one-command demo launcher:

uv run demo.py

You'll need Docker Desktop and uv. A Groq API key is optional; without one, the demo uses a stub model, so generation is simulated while the cache, retrieval, guard, and routing components remain real.

The repository includes the implementation, evaluation reports, architecture, limitations, and instructions for running the tests.

What I'd love feedback on

I'm particularly interested in feedback from people building RAG applications or LLM infrastructure:

What failure cases would you add to the semantic-cache evaluation?
How would you make the entity guard more robust across domains?
What real-world workload would you use to test whether these savings translate to production?

I'd appreciate technical criticism, benchmark suggestions, and contributions.

DEV Community: Yatin Annam