Same Question, Different Words: A Semantic Cache for Free Model Endpoints

Your free model quota is a countdown timer. Every call burns tokens. Most apps waste them on repeat questions.

Same prompt, same answer, same cost. Nobody notices until the quota hits zero.

I built a semantic cache for my free model server. It catches similar prompts and reuses old answers. Here's the full implementation.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why exact caching fails

Exact cache keys are easy. Hash the prompt, store the response. Done.

But users don't type the same prompt twice. They rephrase. They add punctuation. They swap one word.

"Summarize HTTP 429" and "What does HTTP 429 mean?" are the same question. An exact cache misses that. A semantic cache catches it.

How semantic caching works

Three steps:

1. Embed the prompt into a vector
2. Compare it against stored vectors
3. Return the cached answer if similarity passes a threshold

The embedding captures meaning. Similar prompts land close together in vector space. Cosine similarity measures the distance.

The implementation

Here's the complete cache. It uses sentence-transformers for embeddings and numpy for fast similarity search.

import time
import numpy as np
from sentence_transformers import SentenceTransformer

class SemanticCache:
    def __init__(self, model_name="all-MiniLM-L6-v2", threshold=0.90, ttl=3600, max_entries=500):
        self.model = SentenceTransformer(model_name)
        self.threshold = threshold
        self.ttl = ttl
        self.max_entries = max_entries
        self.dim = self.model.get_sentence_embedding_dimension()
        self.embeddings = np.zeros((0, self.dim))
        self.responses = []
        self.timestamps = []

    def _embed(self, text):
        return self.model.encode(text, normalize_embeddings=True)

    def lookup(self, prompt):
        if len(self.responses) == 0:
            return None
        emb = self._embed(prompt)
        scores = self.embeddings @ emb
        idx = int(np.argmax(scores))
        if scores[idx] >= self.threshold:
            return self.responses[idx]
        return None

    def store(self, prompt, response):
        emb = self._embed(prompt)
        self.embeddings = np.vstack([self.embeddings, emb])
        self.responses.append(response)
        self.timestamps.append(time.time())
        if len(self.responses) > self.max_entries:
            self.embeddings = self.embeddings[1:]
            self.responses.pop(0)
            self.timestamps.pop(0)

The np.vstack on every store is O(n). Fine for 500 entries. For thousands, preallocate a buffer.

Wrapping it around API calls

Here's how to use the cache with httpx. I tested this against a free model endpoint on MonkeyCode.

import httpx

cache = SemanticCache(threshold=0.90)

async def get_completion(client, prompt):
    cached = cache.lookup(prompt)
    if cached is not None:
        return cached, "cache"

    try:
        r = await client.post(
            "https://your-endpoint.example/v1/chat",
            json={"prompt": prompt},
            timeout=15
        )
        r.raise_for_status()
        response = r.text
    except httpx.HTTPError:
        raise  # don't cache failures

    cache.store(prompt, response)
    return response, "live"

The flow is simple:

1. Check cache first
2. Call the server only on a miss
3. Store the fresh response

Choosing the threshold

The threshold controls the tradeoff.

High threshold (0.95+): few false matches, low hit rate. Safe but weak.

Low threshold (0.80): more hits, but wrong answers slip through. Two different questions can share a cached response.

Here's a concrete example. "Explain 429 status code" scores 0.97 against "What is a 429 error?" That's a hit. But it scores 0.61 against "Explain 500 status code." Different errors, different answers. The cache correctly misses.

I start at 0.90 and tune from there. Test with your own prompts.

How to measure your hit rate

Run this experiment:

1. Collect 100 real prompts from your app
2. Run them through the cache with threshold 0.90
3. Count how many hit the cache
4. Manually check the cached answers for correctness

def measure_hit_rate(cache, prompts):
    hits = 0
    for p in prompts:
        if cache.lookup(p) is not None:
            hits += 1
    return hits / len(prompts)

A good hit rate for chat-like traffic is 20-40%. If you're lower, lower the threshold. If you see wrong answers, raise it.

When the cache hurts

Semantic caching is wrong for:

• Time-sensitive answers: stock prices, weather, news. Stale data is worse than no data.
• User-specific responses: "my account" questions. One user's answer is another user's leak.
• Creative tasks: the same prompt can produce many valid answers. Caching kills variety.

The TTL handles staleness. Set it to 60 seconds for fast-changing data. Set it to hours for stable facts.

Who should not use this

Don't build this if your traffic is under 50 requests per day. The cache adds complexity without saving much.

Don't use it for PII. Embeddings can leak information about the prompt. That's a real risk.

Don't skip the threshold tuning. A bad threshold makes the cache useless or dangerous.

The takeaway

Free model quotas are precious. Semantic caching stops you from wasting them on repeat questions.

The implementation is 40 lines. The embedding model runs locally. No extra server needed.

Start with threshold 0.90. Measure your hit rate. Tune from real data.

Your quota is a budget. Spend it once per question, not once per phrasing.

A free server option is enough to reproduce the setup.