Semantic Caching Cuts Token Costs for Repeat LLM Prompts

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Semantic caching cuts token costs on repeat prompts by matching meaning, not exact strings: you store each prompt embedding next to its response and reuse that response when cosine similarity crosses a threshold. On repetitive workloads, that skip is often the difference between a token-limited service that lasts the month and one that runs dry before the weekend.

Why Exact-Match Caches Miss the Duplicates You Pay For

I used to treat caching as a string problem: hash the prompt, look it up, done. That fails as soon as wording changes. A support bot that sees "how do I reset my password" and later "I forgot my password, help" misses an exact-match cache and spends tokens on an answer you already computed. The same leak happens when a nightly job re-reads unchanged config files and asks for a summary in slightly different wrapper text.

Semantic caching compares meaning, not strings. You embed the new prompt, compare it to stored embeddings with cosine similarity, and reuse the stored response when the vectors are close enough. Unlike an exact-match cache, this catches rephrased questions and near-duplicate requests. You can run the pattern on MonkeyCode's free server with its free model access, which currently includes 10 million tokens, and the cache itself needs no external service if you keep the vectors in memory.

A practical comparison:

• Exact-match cache: zero false positives, misses every paraphrase, cheapest lookup.
• Semantic cache: catches paraphrases, needs an embedding call on every request, can return a plausible but wrong answer if the threshold is too low.
• No cache: simplest code, full token cost on every duplicate.

On a token-limited server this is not a minor inefficiency. The second call should be free when the question has not changed in meaning.

Build a Semantic Cache in Three Parts

The implementation has three moving parts. The first is an embedding function, which turns a prompt into a vector. The second is a store, which keeps recent embeddings and responses. The third is a threshold, which decides whether two prompts are close enough to share a response. OpenAI-compatible embeddings work here; point the client at any compatible endpoint.

Here is a compact version that fits in a single file:

import numpy as np
from openai import OpenAI

client = OpenAI()  # point this at any OpenAI-compatible endpoint

class SemanticCache:
    def __init__(self, threshold=0.92):
        self.items = []
        self.threshold = threshold

    def embed(self, text):
        r = client.embeddings.create(model="your-embedding-model", input=text)
        return np.array(r.data[0].embedding)

    def lookup(self, prompt):
        vec = self.embed(prompt)
        for stored_vec, stored_resp in self.items:
            if cosine(vec, stored_vec) >= self.threshold:
                return stored_resp
        return None

    def store(self, prompt, response):
        vec = self.embed(prompt)
        self.items.append((vec, response))

def cosine(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

Use it as a wrapper around the model call:

1. Call lookup(prompt) before you hit the chat model.
2. If it returns a response, skip the model and return immediately.
3. If it returns None, call the model as usual.
4. Call store(prompt, response) only when your application logic marks the prompt as safe to reuse.

cache = SemanticCache(threshold=0.92)
hit = cache.lookup(prompt)
if hit is not None:
    return hit
response = client.chat.completions.create(model="your-chat-model", messages=[...])
cache.store(prompt, response.choices[0].message.content)
return response.choices[0].message.content

Start with a threshold of 0.92 and tune on a small set of real prompts. Too high, and you miss paraphrases. Too low, and you return the wrong answer for genuinely different questions. I treat 0.92 as a starting point, not a constant: log similarity scores for both hits and misses, then move the cutoff until false reuse disappears on your own traffic.

What this cache does not do

This in-memory list is linear search. A few hundred embeddings occupy a few megabytes and stay fast enough for a single process. It is not a production vector database, it does not shard, and it does not survive a restart unless you add persistence yourself. That is intentional: measure duplication first, then decide whether the extra machinery is worth it.

Decide What Must Never Be Cached

The harder problem is knowing when not to cache. A response that mentions a timestamp, a user name, or a live metric must not be reused, because the cached copy will be wrong. Semantic similarity does not know that "what is the queue depth right now" is time-sensitive even if the wording matches last night's prompt.

The simplest rule is to cache only prompts that your application logic marks as safe. I use an explicit allow-list rather than trying to detect freshness from the text:

• Cache: FAQ-style support questions, unchanged-file summaries, "still running" status for a short window.
• Do not cache: code generation (each output is unique), anything that interpolates a user name, live metrics, timestamps, or one-off analysis.
• Never cache: medical, legal, or financial advice. A semantic match can return a plausible but wrong answer, and that is unacceptable when exact correctness matters.

You can also include a freshness field and expire entries after a fixed interval. For example, a status-checking loop can cache the "still running" response for thirty seconds, then force a real model call. Expiration is a timestamp on the stored item, not a smarter embedding. If the data can change, the cache must forget.

We applied this pattern to a nightly summarization service on a free server, and the effect was immediate. The batch job re-read the same configuration files every night, and the summaries of unchanged files were nearly identical. The semantic cache caught those duplicates and cut the token consumption by roughly half. The service also became faster, because a cache hit skips the model call entirely and returns in milliseconds.

Measure Duplication Before You Add Complexity

The first thing you should measure is your duplication rate. A semantic cache adds an embedding call to every request, including unique ones. If almost every prompt is new, you pay that overhead and save nothing.

Do this in order:

1. Log every prompt for a representative day.
2. Hash each prompt and count how many times the same hash appears.
3. If fewer than twenty percent of prompts are exact repeats, a semantic cache will not save you much, because embedding calls add overhead to every unique request.
4. If more than half are repeats, the cache is almost certainly worth the complexity.
5. Optionally, embed a sample of prompts and cluster them to estimate semantic duplication, which is higher than exact-match duplication when users rephrase.

The exact-match hash is a good first approximation. I would not start with clustering; it is extra work that only matters after the hash already shows a real repeat problem.

This approach is not for everyone. If your workload consists of unique prompts, a semantic cache will add latency without saving anything. If your responses are highly personalized, the cache will rarely match, and threshold tuning becomes a distraction. The cache is a tool for repetitive workloads, not a general-purpose optimization.

The free tier of MonkeyCode currently includes 10 million tokens and a free server, and that combination is a good place to measure your real duplication rate. Run your traffic through a cache with logging, and you will see how many prompts are actually repeats. That number tells you whether the cache is worth keeping. Free tiers change, so verify the current terms before you depend on them.

If you want to try it, wrap one repetitive path—status polling or unchanged-file summaries—with the class above, log hit rate and similarity scores for a few days, and only then decide whether to keep it. Measure first, cache second, and never reuse an answer that has to be true right now.