Caching in RAG Systems: What to Cache, What Not To, and Why It Matters More Than You Think

Caching is one of the highest-leverage optimizations in a production RAG system and one of the most underused. Most teams cache at the obvious layer, the final LLM response, and miss the more valuable caching opportunities earlier in the pipeline.

Let me walk through the full caching picture for a RAG system, because the right answer is different at each layer.

The embedding layer is where caching has the clearest value proposition. Computing embeddings is deterministic: the same text run through the same embedding model always produces the same vector. Every time a user submits a query you have seen before, you are paying to compute an embedding you already have.

In practice, enterprise RAG systems see high query repetition. Employees ask similar questions. "What is our PTO policy," "how do I submit a reimbursement," "what are the Q3 targets" get asked by many different people. Caching query embeddings means these repeated queries pay the embedding cost once.

import hashlib
import json
from functools import lru_cache

class CachedEmbedder:
    def __init__(self, embedding_model, cache_store):
        self.model = embedding_model
        self.cache = cache_store

    def embed(self, text: str) -> list:
        cache_key = f"emb:{hashlib.md5(text.encode()).hexdigest()}"
        cached = self.cache.get(cache_key)
        if cached:
            return json.loads(cached)

        embedding = self.model.embed(text)
        self.cache.set(cache_key, json.dumps(embedding), ttl=86400)  # 24 hour TTL
        return embedding

The TTL matters here. Embedding models are versioned, and if you upgrade your embedding model, cached embeddings from the old model will be wrong. Either include the model version in the cache key or set a TTL that expires before you expect to update the model.

The retrieval result layer is where most teams try to cache and often get it wrong. Retrieval results are not purely deterministic. The same query will return different results if the underlying document corpus has changed. If you cache retrieval results without accounting for corpus changes, users get stale results.

The correct approach is cache invalidation tied to document updates rather than time-based TTL.

class RetrievalCache:
    def __init__(self, cache_store, document_registry):
        self.cache = cache_store
        self.registry = document_registry

    def get_cache_key(self, query_embedding: list) -> str:
        embedding_hash = hashlib.md5(str(query_embedding).encode()).hexdigest()
        corpus_version = self.registry.get_current_corpus_version()
        return f"retrieval:{embedding_hash}:{corpus_version}"

    def get(self, query_embedding: list):
        key = self.get_cache_key(query_embedding)
        return self.cache.get(key)

    def set(self, query_embedding: list, results: list):
        key = self.get_cache_key(query_embedding)
        self.cache.set(key, results, ttl=3600)

The corpus version is a hash or incrementing counter that changes whenever any document is added, updated, or removed. When the corpus changes, all retrieval cache keys that include the old version become invalid automatically, without needing explicit cache invalidation logic.

The LLM response layer is the most expensive but also the most dangerous to cache. LLM responses are generated given a specific context at a specific time. Caching them means users may get responses that were accurate when generated but are stale now.

My general rule is to only cache LLM responses for queries where the answer is stable over the cache duration. Static reference information, definitions, historical facts. Not policy questions, not questions about current state, not anything where the correct answer might change.

class ConditionalResponseCache:
    CACHEABLE_QUERY_TYPES = {"definition", "historical", "reference"}
    RESPONSE_TTL = {
        "definition": 604800,   # 7 days
        "historical": 2592000,  # 30 days
        "reference": 86400,     # 1 day
    }

    def should_cache(self, query_type: str, context_freshness_days: int) -> bool:
        if query_type not in self.CACHEABLE_QUERY_TYPES:
            return False
        if context_freshness_days > 30:
            return False  # don't cache if source docs are stale
        return True

One caching opportunity that teams consistently miss is prompt prefix caching. If your system prompt is large and constant across all requests, you are paying to process it on every single request. Both Anthropic and OpenAI support prompt caching that charges significantly reduced rates for the cached portion. On a high-volume deployment, this can represent 15 to 25% reduction in inference cost for zero change in functionality.

The only requirement is that the cacheable content appears at the beginning of the prompt in the same position across requests. If your system prompt is dynamically assembled with session-specific content inserted before the stable portion, restructure the prompt so the stable content comes first.

Caching done well makes your RAG system faster and cheaper without degrading quality. Caching done poorly gives users confidently wrong answers from stale cache entries. The difference is designing each caching layer around the specific properties of the data being cached rather than applying a generic caching strategy across the whole pipeline.