Stop Paying Twice for the Same Answer: A Semantic Cache for LLM Calls

The most expensive token is the one you spend on a question you have already answered. My position is that semantic caching should be the first layer in any pipeline that runs on a free model quota, because it converts repeated work into a single paid request. This article walks through a working semantic cache, the threshold math behind it, and the failure modes you need to respect.

MonkeyCode is an open-source project with a free tier: ten million tokens and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The cache pattern below works with any endpoint, and the free tier simply makes the savings visible sooner.

The hidden cost of repeated prompts

Most pipelines send the same semantic question to a model many times, with slightly different wording each time. A user asks how to fix an error, then asks what causes the same exception, and the model burns two completions on what is essentially one answer. Exact-match caching catches none of this, because the strings differ even when the meaning does not.

The fix is a semantic cache: an embedding-based lookup that decides whether a new prompt is close enough to a stored one to reuse the old answer. When the cache hits, the token cost drops to zero, because the answer comes from storage rather than generation. When it misses, you pay for a single completion and store the result for the next similar question.

How the cache works

The pipeline has four numbered steps:

1. Embed the incoming prompt with a cheap embedding model.
2. Compare the embedding against stored entries using cosine similarity.
3. Return the cached answer when similarity exceeds a threshold.
4. Store the new prompt and answer when it does not.

The embedding step is the only part that costs tokens, and embedding models are dramatically cheaper than generation models. A cache hit costs one embedding call, which is a rounding error compared to a full completion.

The artifact: a semantic cache in Python

Here is the complete implementation, written for Python 3.11 with the standard library plus sentence-transformers and numpy:

#!/usr/bin/env python3
"""Semantic cache for LLM calls with cosine similarity."""
import sqlite3
from pathlib import Path

import numpy as np
from sentence_transformers import SentenceTransformer

MODEL_NAME = "all-MiniLM-L6-v2"  # ~80MB, runs on CPU
SIMILARITY_THRESHOLD = 0.85
DB_PATH = Path("semantic_cache.db")


class SemanticCache:
    def __init__(self, model_name=MODEL_NAME, threshold=SIMILARITY_THRESHOLD):
        self.model = SentenceTransformer(model_name)
        self.threshold = threshold
        self.conn = sqlite3.connect(str(DB_PATH))
        self.conn.execute("""
            CREATE TABLE IF NOT EXISTS cache (
                id INTEGER PRIMARY KEY,
                prompt TEXT NOT NULL,
                embedding BLOB NOT NULL,
                answer TEXT NOT NULL,
                created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
            )
        """)
        self.conn.commit()

    def _embed(self, text):
        vector = self.model.encode(text, normalize_embeddings=True)
        return vector.astype(np.float32).tobytes()

    def _cosine_similarity(self, a_bytes, b_bytes):
        a = np.frombuffer(a_bytes, dtype=np.float32)
        b = np.frombuffer(b_bytes, dtype=np.float32)
        return float(np.dot(a, b))

    def get(self, prompt):
        query_embedding = self._embed(prompt)
        rows = self.conn.execute("SELECT prompt, embedding, answer FROM cache").fetchall()
        best_score = 0.0
        best_answer = None
        for stored_prompt, stored_embedding, answer in rows:
            score = self._cosine_similarity(query_embedding, stored_embedding)
            if score > best_score:
                best_score = score
                best_answer = answer
        if best_score >= self.threshold:
            return {"hit": True, "answer": best_answer, "score": round(best_score, 3)}
        return {"hit": False, "score": round(best_score, 3)}

    def put(self, prompt, answer):
        embedding = self._embed(prompt)
        self.conn.execute(
            "INSERT INTO cache (prompt, embedding, answer) VALUES (?, ?, ?)",
            (prompt, embedding, answer),
        )
        self.conn.commit()


def cached_completion(prompt, llm_call, cache):
    """Wrap any LLM call with the semantic cache."""
    result = cache.get(prompt)
    if result["hit"]:
        return result["answer"]
    answer = llm_call(prompt)
    cache.put(prompt, answer)
    return answer

The cache stores prompts, embeddings, and answers in SQLite, which keeps the whole system to a single file. The cached_completion wrapper is the integration point: swap it into any existing pipeline without touching the rest of the code. The similarity threshold of 0.85 is a starting point, not a law, and the next section explains how to tune it.

Threshold tuning: the tradeoff that matters

The threshold controls the balance between false hits and wasted tokens, and finding the right value is a measurement exercise, not a guess.

Step one: collect a sample of real prompts. Take fifty prompts from your actual usage, and manually label which pairs should share an answer.

Step two: compute pairwise similarities. Run all pairs through the embedding model and record the scores for true-similar and true-different pairs.

Step three: pick the threshold at the gap. Look for the score range where similar pairs and different pairs separate, and set the threshold in that gap.

A threshold that is too low returns wrong answers, which is worse than paying for a new completion. A threshold that is too high turns the cache into an exact-match store, which defeats the purpose.

Measuring the hit rate

A semantic cache without a hit-rate measurement is a guess with a database attached, so instrument the wrapper from day one. Add a counter to the integration point and log the ratio after every run:

cache_stats = {"hits": 0, "misses": 0}

def cached_completion_with_stats(prompt, llm_call, cache):
    result = cache.get(prompt)
    if result["hit"]:
        cache_stats["hits"] += 1
        return result["answer"]
    cache_stats["misses"] += 1
    answer = llm_call(prompt)
    cache.put(prompt, answer)
    return answer

Track the hit ratio over a full week and compare it against your threshold to learn whether the cache is actually saving tokens. A hit rate below twenty percent means your prompts are too unique for semantic caching, and you should lower the threshold or abandon the pattern. A hit rate above sixty percent means the cache is earning its keep, and you should invest in better eviction and more storage.

Eviction and invalidation

Cached answers go stale when the underlying code or data changes, and a semantic cache needs an eviction policy. The simplest policy is time-based: delete entries older than N days with a single SQL statement. The next simplest is capacity-based: keep the most recently used entries and drop the rest.

-- Time-based eviction: delete entries older than 30 days
DELETE FROM cache WHERE created_at < datetime('now', '-30 days');

-- Capacity-based eviction: keep the 1000 most recent entries
DELETE FROM cache WHERE id NOT IN (
    SELECT id FROM cache ORDER BY created_at DESC LIMIT 1000
);

Run either statement on a schedule, or trigger it after a deploy that changes the domain. Invalidation is the part most teams skip, and it is the part that turns a helpful cache into a source of confident wrong answers.

Decision table: where semantic caching helps

Workload	Cache benefit	Reason
Support-style Q&A over docs	High	Users ask the same questions with different wording
Code review explanations	Medium	Similar diffs appear, but each one has unique context
One-off creative generation	Low	Every prompt is genuinely new
Summarization of unique documents	None	No repeated semantic content
Interactive chat with long context	None	Context makes each request unique

The pattern is simple: semantic caching pays when your users ask the same questions repeatedly, and it does nothing when every prompt is genuinely novel.

Limitations and who should skip this

The embedding model adds a dependency and a small latency cost on every request, even cache hits. The cache stores prompts and answers in plain SQLite, so do not use it for sensitive data unless you encrypt the database. The similarity threshold needs periodic re-evaluation as your prompt distribution shifts. Teams that serve highly unique, context-dependent queries should skip caching entirely, because the miss rate will never justify the overhead.

The conclusion that pays for itself

A semantic cache turns a free token budget into a reusable resource, because the second ask of the same question costs nothing. Build the cache, tune the threshold, and watch your completion count drop while your answer coverage stays flat. If you want to see the pattern under real quota pressure, MonkeyCode's free tier gives you enough tokens to measure the hit rate yourself.