I Built a 3-Layer Prompt Cache and Cut My LLM Costs by 73%

I was burning through $200/month on LLM API calls. The worst part? I knew at least half of those requests were sending nearly identical prompts over and over.

So I built a cache. Not a Redis one-liner — a three-layer system that catches duplicates at different levels. Here's how it works, and the numbers I got.

The Problem

Every time a user asks "Summarize this document," my backend sends the full system prompt (500+ tokens of instructions) plus the document text to the LLM. Multiply that by 100 requests per hour, and you're paying to send the same system prompt 100 times.

But it's worse than that. Users ask similar questions. "Summarize this" and "Give me a summary" and "TLDR this" are semantically identical. An exact-match cache won't catch those.

And even when the question is totally different, the structure of my prompts is the same — same few-shot examples, same output format instructions, same "you are a helpful assistant" preamble. That's 200+ tokens I'm paying for on every single call.

The Three Layers

I broke caching into three tiers, from cheapest to most expensive:

Layer 1: Exact Match Cache (LRU)

This is your standard key-value cache. Hash the full prompt, check if we've seen it before.

import hashlib
from collections import OrderedDict

class ExactCache:
    def __init__(self, max_size=10000):
        self.cache = OrderedDict()
        self.max_size = max_size

    def _hash(self, prompt: str) -> str:
        return hashlib.sha256(prompt.encode()).hexdigest()

    def get(self, prompt: str):
        key = self._hash(prompt)
        if key in self.cache:
            self.cache.move_to_end(key)  # LRU bump
            return self.cache[key]
        return None

    def set(self, prompt: str, response: str):
        key = self._hash(prompt)
        self.cache[key] = response
        self.cache.move_to_end(key)
        if len(self.cache) > self.max_size:
            self.cache.popitem(last=False)

This catches about 15% of requests in my system — mostly identical queries from different users hitting the same endpoint.

Layer 2: Semantic Cache (Embedding + Cosine)

This is where things get interesting. Instead of requiring an exact match, we compare the meaning of prompts using embeddings.

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

class SemanticCache:
    def __init__(self, embed_fn, threshold=0.95, max_size=5000):
        self.embed = embed_fn  # your embedding function
        self.threshold = threshold
        self.entries = []  # list of (vector, prompt, response)
        self.max_size = max_size

    def get(self, prompt: str):
        vec = self.embed(prompt)
        if not self.entries:
            return None

        vectors = np.array([e[0] for e in self.entries])
        similarities = cosine_similarity([vec], vectors)[0]
        best_idx = np.argmax(similarities)

        if similarities[best_idx] >= self.threshold:
            return self.entries[best_idx][2]  # response

        return None

    def set(self, prompt: str, response: str):
        vec = self.embed(prompt)
        self.entries.append((vec, prompt, response))
        if len(self.entries) > self.max_size:
            self.entries.pop(0)

I use a lightweight embedding model (all-MiniLM-L6-v2, runs in ~10ms on CPU). Setting the threshold at 0.95 means "Summarize this report" and "Give me a summary of this report" match, but "Write a poem about this report" doesn't. This catches another 22% of requests.

A warning: if your threshold is too low (say 0.85), you'll get false positives — semantically similar but meaningfully different queries getting cached responses. I started at 0.90 and crept up to 0.95 after a few embarrassing incidents.

Layer 3: Prompt Template Cache

Most LLM applications use structured prompts with a fixed template and variable inputs. Instead of sending the full prompt every time, we leverage the provider's prompt caching features. Anthropic's prompt caching saves 90% on the cached portion of the prompt, and many OpenAI-compatible providers offer similar mechanisms.

class TemplateCache:
    def __init__(self, llm_call_fn):
        self.llm_call = llm_call_fn

    def call(self, system_prompt: str, user_input: str):
        # Anthropic: mark system as cache_control block
        # OpenAI-compatible: system message benefits from
        # automatic prefix caching on many providers
        return self.llm_call(system_prompt, user_input)

This layer catches the remaining 36% of token spend — those calls where the specific user input is unique, but the system prompt and template structure are identical to thousands of previous calls.

The Pipeline

All three layers in sequence:

class PromptCachePipeline:
    def __init__(self, embed_fn, llm_call_fn):
        self.exact = ExactCache(max_size=10000)
        self.semantic = SemanticCache(embed_fn, threshold=0.95)
        self.template = TemplateCache(llm_call_fn)

    def call(self, system_prompt: str, user_input: str):
        full_prompt = f"{system_prompt}\n\n{user_input}"

        # Layer 1: Exact match
        cached = self.exact.get(full_prompt)
        if cached:
            return cached, "exact_hit"

        # Layer 2: Semantic match
        cached = self.semantic.get(full_prompt)
        if cached:
            self.exact.set(full_prompt, cached)  # promote to exact
            return cached, "semantic_hit"

        # Layer 3: Call LLM
        response = self.template.call(system_prompt, user_input)

        # Store in all layers
        self.exact.set(full_prompt, response)
        self.semantic.set(full_prompt, response)

        return response, "miss"

Real Numbers

I ran this on a system handling ~50,000 LLM calls per day across five different prompt types (summarization, extraction, classification, Q&A, code review).

Layer	Hit Rate	Token Savings
Exact Cache	15.2%	2.1M tokens/day
Semantic Cache	22.4%	3.8M tokens/day
Template Cache	35.8%	7.2M tokens/day
Total	73.4%	13.1M tokens/day

At GPT-4o pricing ($2.50/1M input tokens), that's about $33/day saved, or $990/month. On a system that was costing $1,350/month, the new bill is roughly $360.

What I'd Do Differently

Three things I learned the hard way:

1. Eviction strategy matters. I started with pure LRU for the semantic cache — bad idea. An LRU cache evicts the least recently used entry, but in an LLM system, some prompts are rare but expensive (long documents). I switched to a weighted eviction that considers both recency and token count. A 2000-token cached response is worth more than a 50-token one.

2. Embedding model choice is critical. I tried OpenAI's text-embedding-3-small first — great quality, but the API latency killed my response times (adding 80-120ms). Switching to a local model (all-MiniLM-L6-v2 via sentence-transformers) brought that down to 8-12ms. The quality difference at a 0.95 threshold was negligible.

3. Don't cache everything. Some prompts should never be cached — anything with time-sensitive data ("what's the weather"), user-specific context ("based on my previous message"), or creative generation ("write a unique story about..."). I added a simple no_cache flag to my pipeline that certain endpoints set.

Try It Yourself

The full implementation (with tests and the weighted eviction strategy) is about 200 lines of Python. You can wire it into an existing LLM pipeline in an afternoon. Start with Layer 1 — the exact match cache — and you'll probably recoup the implementation time in API savings within the first week.

---

The biggest surprise wasn't the cost savings. It was the latency improvement. A cache hit in Layer 1 returns in under 1ms instead of 800ms+ for an LLM round-trip. Users noticed. Your mileage will vary depending on how repetitive your prompt patterns are, but if you're running any kind of production LLM service, I'd bet at least 30% of your prompts are cacheable.