The cheapest model call is the one that never leaves the process. On a zero-budget stack, the real risk is not an invoice — it is burning a metered allowance on work that was already computed one restart ago.
This is a build-along for a single-founder setup: one app, one free host, one small model budget. The artifact is a content-addressed cache in stdlib Python plus SQLite, small enough to read in a sitting, durable enough to survive the container being recycled overnight.
Why an in-process dict dies on a free host
A dict cache is correct and useless here. Free hosts sleep, redeploy, and restart on their own schedule. Every cycle empties memory, and the next identical request is billed again.
The failure repeats in three places: a retry after a timeout, a double-submitted form, and a background job that re-runs on boot after a dirty shutdown. Each one sends the same prompt and gets the same answer for the second time.
Two conditions make this sharp on a zero-cost stack: free model access that is metered, and a free server option that can restart without notice. Both are operator-supplied availability claims from the MonkeyCode project, not guarantees of uptime or quota. Verify current terms before depending on them.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Step 1 — Hash the request, not the user
The key must describe everything that changes the output: model identifier, prompt text, and sampling parameters. Canonical JSON with sorted keys makes the hash stable across processes.
import hashlib, json
def cache_key(model: str, prompt: str, params: dict) -> str:
payload = json.dumps(
{"model": model, "prompt": prompt, "params": params},
sort_keys=True, separators=(",", ":"),
).encode("utf-8")
return hashlib.sha256(payload).hexdigest()
Because the model id and parameters are inside the hash, a model swap or a temperature change invalidates the entry automatically. There is no manual flush step to forget.
Step 2 — One table, one file, one writer
SQLite is enough for a solo project and it survives a process restart. WAL mode keeps reads from blocking the single writer.
import sqlite3, time
class GenCache:
def __init__(self, path="gen_cache.sqlite3",
ttl_s=7 * 24 * 3600, max_bytes=32 * 1024 * 1024):
self.ttl_s, self.max_bytes = ttl_s, max_bytes
self.db = sqlite3.connect(path)
self.db.execute("PRAGMA journal_mode=WAL")
self.db.execute("""
CREATE TABLE IF NOT EXISTS gen_cache (
key TEXT PRIMARY KEY,
created_at REAL NOT NULL,
bytes INTEGER NOT NULL,
output TEXT NOT NULL
)""")
self.db.commit()
def get(self, key: str):
row = self.db.execute(
"SELECT created_at, output FROM gen_cache WHERE key=?", (key,)
).fetchone()
if row is None:
return None
created_at, output = row
if time.time() - created_at > self.ttl_s:
self.db.execute("DELETE FROM gen_cache WHERE key=?", (key,))
self.db.commit()
return None
return output
def put(self, key: str, output: str) -> None:
n = len(output.encode("utf-8"))
self.db.execute(
"INSERT OR REPLACE INTO gen_cache(key, created_at, bytes, output)"
" VALUES (?,?,?,?)", (key, time.time(), n, output))
self.db.commit()
self._trim()
def _trim(self) -> None:
total = self.db.execute(
"SELECT COALESCE(SUM(bytes), 0) FROM gen_cache").fetchone()[0]
while total > self.max_bytes:
row = self.db.execute(
"SELECT key, bytes FROM gen_cache"
" ORDER BY created_at ASC LIMIT 1").fetchone()
if row is None:
break
self.db.execute("DELETE FROM gen_cache WHERE key=?", (row[0],))
total -= row[1]
self.db.commit()
The byte ceiling matters more than row count. A few long documents can consume the disk allowance on a small free instance, and eviction by age is the cheapest policy that behaves predictably.
Step 3 — Wrap the call so failures never get cached
A cache miss should be the only path to the model, and only a non-empty result should be written back. Caching an error turns a transient failure into a permanent wrong answer.
def generate(prompt, model, params, call_model, cache):
key = cache_key(model, prompt, params)
hit = cache.get(key)
if hit is not None:
return hit, "hit"
out = call_model(model=model, prompt=prompt, **params) # provider-specific
if out: # do not memoize empty or failed responses
cache.put(key, out)
return out, "miss"
Note the label: returning "hit" or "miss" alongside the text costs nothing and gives a counter to log. It is the difference between guessing whether the cache works and knowing.
Step 4 — A test plan that needs no API key
Run this before wiring the real client. A counting fake model proves the behavior offline in under a second.
calls = []
def fake_model(model, prompt, **params):
calls.append(prompt)
return "" if prompt == "fail" else f"echo:{prompt}"
cache = GenCache(path=":memory:")
out, state = generate("summarize a", "m1", {"temperature": 0}, fake_model, cache)
assert state == "miss" and len(calls) == 1
out, state = generate("summarize a", "m1", {"temperature": 0}, fake_model, cache)
assert state == "hit" and len(calls) == 1 # no second call
out, state = generate("summarize a", "m1", {"temperature": 0.7}, fake_model, cache)
assert state == "miss" and len(calls) == 2 # params bust the key
out, state = generate("fail", "m1", {}, fake_model, cache)
assert state == "miss"
out, state = generate("fail", "m1", {}, fake_model, cache)
assert state == "miss" and len(calls) == 4 # nothing cached for empty output
cache.put(cache_key("m1", "x", {}), "y")
cache.ttl_s = -1
assert cache.get(cache_key("m1", "x", {})) is None # expiry path
The last two assertions are the ones worth keeping forever. Caching empty strings hides hard failures, and an untested expiry path is how stale text reaches production.
When to cache and when to refuse
| Request shape | Cache? | Key additions | TTL |
|---|---|---|---|
| Extract fields from the same document | Yes | document hash | 30 days |
| Summarize a support thread | Yes | thread id | 7 days |
| Creative draft, high temperature | Yes, cautiously | temperature, seed | hours |
| Per-user personalized answer | Only if scoped | user id in key | minutes |
| Anything containing secrets or PII | Not by default | redact first | n/a |
| Streaming responses | Cache the final text | prompt only | per case |
Two rules follow from the table. First, scope personalization into the key or leave it uncached — a shared entry leaking one user's context into another's answer is worse than any bill. Second, strip sensitive material before it touches disk, since the SQLite file is plaintext.
Limitations worth stating up front
A cache does not reduce the cost of the first call. Applications dominated by unique prompts will see a low hit rate and should not add this layer at all.
The free server option can reset the filesystem along with the process. If the SQLite file does not live on a mounted path, the cache is a best-effort optimization that may vanish between deploys — which is acceptable, because it holds derived data, never source-of-truth records.
Staleness is the other edge. A short TTL keeps answers close to the current model behavior; a long TTL keeps the hit rate high. That trade is a product decision, not a technical one, and it belongs in the config, visible in one place.
Who should skip this
Skip it for one-off scripts, for workloads with per-user-unique prompts, for regulated data that cannot be written to local plaintext storage, and for any system where a wrong cached answer would be harder to debug than a slow one. Also skip it if the app already sits behind a CDN cache that covers the same requests.
For the solo founder shipping today, the sequence is short: add the key function, add the table, count hits, then stop thinking about it. Readers who want to try the workflow without provisioning anything can start on the project's free model access and free server option, and should check the current terms before relying on either for anything load-bearing.
Top comments (1)
This is why I think the interface matters as much as the model.