Monday at 9:30 AM, the CI queue runs three deep. Your review bot processes the same pull request for the fourth time. The prompt carries yesterday's context, and the model returns the same verdict. The token counter ticks up anyway.
Free token grants look generous at first glance. Ten million tokens sounds infinite until your team shares one endpoint. The grant shrinks faster than anyone expects, and repetition is the silent killer.
MonkeyCode offers free model access and a free hosted server. The allocation is real, but the spending discipline is on you.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Before you build on a free tier, build a cache. The cheapest LLM call is the one you never make.
The repetition problem
LLM calls are not all unique in practice. CI reviews repeat the same diff many times, and test generators rerun identical scenarios. Support bots answer the same question twice per hour, and each duplicate burns tokens without adding any value.
A cache turns duplicates into free hits instantly. The first call pays the full price, and the next hundred calls pay nothing.
Three layers of caching
Exact matching is the cheapest layer to implement. Hash the normalized prompt and compare hashes, and zero model calls are needed for a match.
Normalized matching handles formatting noise without extra cost. Extra spaces and reordered keys disappear, and same meaning produces the same hash.
Semantic matching catches paraphrases, but it needs an embedding call. It costs tokens, so use it only when the first two layers miss.
"""A layered response cache for OpenAI-compatible chat endpoints."""
import hashlib
import json
import os
import re
import time
from collections import OrderedDict
import requests
BASE_URL = os.environ["OPENAI_BASE_URL"].rstrip("/")
API_KEY = os.environ["OPENAI_API_KEY"]
MODEL = os.environ["MODEL_NAME"]
CACHE_SIZE = 256
TTL_SECONDS = 3600
def normalize_prompt(messages):
"""Normalize whitespace and key order for stable hashing."""
def clean(text):
return re.sub(r"\s+", " ", text.strip())
return json.dumps(
[{"role": m["role"], "content": clean(m["content"])} for m in messages],
sort_keys=True,
)
class ResponseCache:
def __init__(self, capacity=CACHE_SIZE, ttl=TTL_SECONDS):
self.capacity = capacity
self.ttl = ttl
self._store = OrderedDict()
def _key(self, messages):
raw = normalize_prompt(messages)
return hashlib.sha256(raw.encode()).hexdigest()
def get(self, messages):
key = self._key(messages)
if key not in self._store:
return None
entry = self._store[key]
if time.time() - entry["ts"] > self.ttl:
del self._store[key]
return None
self._store.move_to_end(key)
return entry["response"]
def put(self, messages, response):
key = self._key(messages)
self._store[key] = {"ts": time.time(), "response": response}
self._store.move_to_end(key)
while len(self._store) > self.capacity:
self._store.popitem(last=False)
cache = ResponseCache()
def chat_completion(messages):
cached = cache.get(messages)
if cached:
return cached
resp = requests.post(
f"{BASE_URL}/chat/completions",
headers={"Authorization": f"Bearer {API_KEY}"},
json={"model": MODEL, "messages": messages, "stream": False},
timeout=60,
)
resp.raise_for_status()
body = resp.json()
cache.put(messages, body)
return body
The cache key is a SHA-256 hash of the normalized prompt. Normalization matters more than the hash function itself. Whitespace, key order, and role labels all affect the hash, so the normalizer strips and sorts them away.
Temperature and max_tokens also change responses. Include them in the cache key if you vary them. A cached response for temperature 0.7 is wrong for temperature 0.2.
Measuring hit rate
A cache without a hit-rate metric is a guess. Add a counter subclass to track hits and misses.
class CountingCache(ResponseCache):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.hits = 0
self.misses = 0
def get(self, messages):
result = super().get(messages)
if result is not None:
self.hits += 1
else:
self.misses += 1
return result
@property
def hit_rate(self):
total = self.hits + self.misses
return self.hits / total if total else 0.0
Run it against a real workload and record the hit rate. Then decide if semantic matching is worth the embedding cost.
A useful test workload has three distinct parts. The mix includes exact duplicates, near duplicates, and unique prompts. A realistic mix mirrors how your team actually calls the API.
| Workload mix | Expected hit rate | Action |
|---|---|---|
| 60% duplicates | above 50% | Keep exact cache |
| 30% near-duplicates | 10-30% | Add normalization |
| 10% unique | below 10% | Accept the misses |
These numbers are starting points, so measure your own traffic.
Concurrency and staleness
The cache is not thread-safe by default. Two threads can miss simultaneously and issue duplicate calls. A double-checked lock prevents that waste.
import threading
lock = threading.Lock()
def chat_completion_threadsafe(messages):
cached = cache.get(messages)
if cached:
return cached
with lock:
cached = cache.get(messages)
if cached:
return cached
resp = requests.post(
f"{BASE_URL}/chat/completions",
headers={"Authorization": f"Bearer {API_KEY}"},
json={"model": MODEL, "messages": messages, "stream": False},
timeout=60,
)
resp.raise_for_status()
body = resp.json()
cache.put(messages, body)
return body
The lock is held during the check-and-set window. The HTTP request happens inside the lock, which serializes duplicate calls. That is acceptable because the duplicates are the waste you are eliminating.
A cached response is also a snapshot of the past. Models change and contexts drift, so cached answers go stale. Set a TTL that matches your data freshness. One hour works for code review, and one minute works for live data. Anything time-critical should use a zero TTL.
The cache hides quality regressions too. If the model improves, your cache serves the old answer. Flush the cache after every model update.
Who should skip this
Personal projects with low traffic do not need a cache. The complexity is not worth the token savings.
Applications with unique prompts get no benefit at all. If every call is different, the cache is dead weight.
Security-sensitive workloads should never cache model responses. Prompt injection can poison a cache, and a poisoned response spreads fast.
The habit
Build the cache before you optimize anything else. Token grants are finite by design, and repetition is the easiest waste to eliminate.
On a hosted free server, latency is often higher than a paid endpoint. A cache reduces round trips, which makes the tool feel faster without server-side changes.
MonkeyCode's free tier is a reasonable place to practice this discipline. The cache works with any OpenAI-compatible endpoint, and the habit outlasts the grant.
Top comments (0)