Stretch a Free LLM Quota With a 40-Line Semantic Cache

Your free tier isn't dying from unique prompts. It's dying from the same question asked forty different ways.

Don't believe me? Run a duplicate check on your own logs. Embed your last ten thousand prompts, cluster them by cosine similarity, and count how many have a semantic twin. In most support, QA, and code-review workloads, that number lands somewhere between 40% and 70%.

Everyone's talking about memory for LLMs this week — reasoning ledgers, agent state, vector stores. The cheapest memory layer you can add is the one that remembers what you already answered. A semantic cache. Forty lines of Python. I'll show you the whole thing, then I'll show you where it lies to you.

Step 1: Measure your duplicate rate first

Don't build the cache yet. Measure first.

1. Export your last 10,000 real prompts. Not test prompts — real ones.
2. Embed them with any embedding model.
3. Compute pairwise cosine similarity.
4. Count prompts with at least one neighbor above 0.93.

Here's the script:

import numpy as np
from openai import OpenAI

client = OpenAI()
prompts = [p.strip() for p in open("prompts.txt") if p.strip()]

embs = np.array([
    np.array(client.embeddings.create(
        model="embedding-model", input=p
    ).data[0].embedding, dtype="float32")
    for p in prompts
])

embs /= np.linalg.norm(embs, axis=1, keepdims=True)
sim = embs @ embs.T
np.fill_diagonal(sim, 0)

has_twin = (sim >= 0.93).any(axis=1)
print(f"{has_twin.sum()} of {len(prompts)} prompts have a semantic twin")
print(f"duplicate ratio: {has_twin.mean():.0%}")

Run it:

python3 duplicate_check.py

If your ratio is under 10%, stop reading. A cache won't save you. If it's above 30%, keep going — the cache pays for itself within a week.

Step 2: How a semantic cache works

The idea is simple. Every prompt becomes a vector. When a new prompt arrives, you compare it against everything you've already answered. If the closest match clears a threshold, you return the old answer instead of calling the model.

So what's the right threshold? That's the whole game.

Too high, and every prompt misses — you save nothing. Too low, and the cache starts answering questions it shouldn't. "Is the payment API down?" and "Is the payment API up?" share almost every word. Their embeddings sit close together. But the answers are opposites. A threshold of 0.93 handles that. A threshold of 0.80 doesn't.

Step 3: The proxy code

Here's the complete proxy. FastAPI, one dictionary, no database, no Redis, no orchestration.

# cache_proxy.py
import numpy as np
from fastapi import FastAPI, Request
from openai import OpenAI

client = OpenAI()  # point base_url at any OpenAI-compatible endpoint
app = FastAPI()
store = {}  # embedding bytes -> (embedding, answer)

def embed(text: str):
    r = client.embeddings.create(model="embedding-model", input=text)
    return np.array(r.data[0].embedding, dtype="float32")

def similarity(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

@app.post("/chat")
async def chat(req: Request):
    body = await req.json()
    prompt = body["prompt"]
    threshold = float(body.get("threshold", 0.93))

    emb = embed(prompt)
    for key, (cached_emb, cached_answer) in store.items():
        if similarity(emb, cached_emb) >= threshold:
            return {"answer": cached_answer, "hit": True}

    resp = client.chat.completions.create(
        model=body.get("model", "default"),
        messages=[{"role": "user", "content": prompt}],
    )
    answer = resp.choices[0].message.content
    store[emb.tobytes()] = (emb, answer)
    return {"answer": answer, "hit": False}

Run it:

pip install fastapi uvicorn openai numpy
uvicorn cache_proxy:app --port 8000

That's the whole thing. One endpoint, one dictionary, one threshold.

Step 4: Prove it works

Now hit it with five prompts that ask the same thing differently:

# probe.py
import json, time, urllib.request

prompts = [
    "How do I paginate in Django?",
    "What's the best way to paginate Django querysets?",
    "Django pagination, how does it work?",
    "Explain Django's Paginator class",
    "How do I split a queryset into pages in Django?",
]

for p in prompts:
    data = json.dumps({"prompt": p}).encode()
    req = urllib.request.Request(
        "http://localhost:8000/chat", data=data,
        headers={"Content-Type": "application/json"},
    )
    t0 = time.time()
    resp = json.load(urllib.request.urlopen(req))
    print(f"hit={resp['hit']}  {time.time() - t0:.2f}s  {p[:45]}")

Expected output:

hit=False  1.20s  How do I paginate in Django?
hit=True   0.05s  What's the best way to paginate Django querysets?
hit=True   0.04s  Django pagination, how does it work?
hit=True   0.05s  Explain Django's Paginator class
hit=True   0.04s  How do I split a queryset into pages in Django?

Four model calls saved. Latency dropped from 1.2 seconds to 50 milliseconds. That's the trade: a little spent on embeddings, a lot saved on generation.

Step 5: Where the cache lies to you

The cache is dumb. It doesn't understand prompts. It only knows that two vectors point in similar directions. That causes four specific failures:

1. The negation trap. "Is the API down?" and "Is the API up?" are 0.95 similar and logically opposite. Your threshold needs to be high enough to separate them.
2. Stale answers. The model gets updated, but the cache still holds yesterday's answer. For volatile topics, add a TTL and expire entries after a few hours.
3. Memory growth. The store grows forever. In production, swap the dict for an LRU cache and cap it at a few thousand entries.
4. Embedding cost. Every request now costs an embedding call. If your embedding model is pricey and your prompts are short, you can spend more on embeddings than you save on generation.

None of these are fatal. But they're why the threshold belongs in a config file, not in a constant.

Who should not use this

Three workloads should skip this pattern entirely:

• Real-time data. Stock prices, deployment status, weather — anything where the answer changes by the minute. A cached answer is a wrong answer.
• Personalized responses. If every answer depends on user context, semantic similarity means nothing. Two users asking the same question need different answers.
• Compliance-heavy pipelines. If you must prove which model version produced which response, an in-memory cache without version tracking won't pass an audit.

For those, pay for the tokens. The cache isn't a replacement for budget. It's a way to stop burning budget on repetition.

Try the whole experiment for free

Here's the part I like about this pattern: the entire experiment costs nothing to run.

MonkeyCode is an open-source project that offers free model access and a free server option. The current allowance includes 10 million free tokens — enough for the duplicate check, the proxy, and a few thousand real prompts.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free server matters here. You get a box to run the proxy on, and the free tokens cover the model calls behind it. No credit card, no GPU, no trial-period countdown. Just the experiment.

Run the duplicate check on your own logs first. If your ratio is above 30%, build the cache and measure the hit rate for a week. Then come back and tell me where your threshold landed — I'm genuinely curious how that number shifts across different workloads.