DEV Community

Casey Zhang
Casey Zhang

Posted on

Build an LLM Cache Proxy in 40 Lines to Cut Token Costs 83%

#llmcache #python #sqlite #tokencost

If your scripts send the same LLM prompt more than once, you are paying twice for the same answer. I built a lightweight SQLite-backed HTTP cache proxy in under 40 lines of Python, ran it on a free server, and cut token spend by 83% on a 214-call batch job.

Why a Caching Proxy Beats an In-Process Cache

You can cache inside the script with a dictionary or a JSON file. That cache dies when the process exits. The moment a cron job finishes, tomorrow's run starts from zero. A proxy is a separate process with a persistent SQLite store, so later runs still reuse earlier answers.

Here is what a proxy gives you that an in-process cache does not:

  • Persistence across runs — SQLite survives process restarts, so Monday's answers are still there on Tuesday.
  • A shared cache for multiple scripts — any HTTP client can hit the same endpoint without sharing a process or a file lock.
  • No changes to your model-call logic — point the client at the proxy URL and keep the same request body.

The tradeoff is one extra network hop. For batch jobs and scheduled tasks that hop is negligible next to token savings.

Compared with a per-script dict, the proxy outlives the job. Compared with Redis or Memcached, SQLite needs no extra daemon, which is why it fits a free server running a single Python process. Skip this pattern if every prompt is unique; you would add latency for zero hits.

I chose exact-match caching on purpose. It is the smallest thing that can prove whether a workload even has enough repetition to bother caching. If the hit rate is high, you already saved money. If it is low, you learned that before writing an embedding pipeline.

How to Build the Proxy with Python's Standard Library

The proxy is a plain HTTP server. Python's standard library is enough — no Flask, no FastAPI, no pip packages. I used http.server for the listener, hashlib for the cache key, and SQLite for storage.

Step 1: Create the server and cache table 

from http.server import BaseHTTPRequestHandler, HTTPServer
import json, hashlib, sqlite3, urllib.request

DB = sqlite3.connect('cache.db', check_same_thread=False)
DB.execute('CREATE TABLE IF NOT EXISTS cache (key TEXT PRIMARY KEY, response TEXT, created_at TEXT DEFAULT CURRENT_TIMESTAMP)')

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers.get('Content-Length', 0))
        body = json.loads(self.rfile.read(length))
        prompt = body.get('messages', [])[-1]['content']
        cache_key = hashlib.sha256(prompt.encode()).hexdigest()
Enter fullscreen mode Exit fullscreen mode

The cache key is a SHA-256 of the last user message — the part of the request that usually determines the answer. If your prompts include system messages or parameters like temperature, fold those into the hash as well. For most exact-match caching, the user message is enough.

check_same_thread=False is required because http.server can invoke the handler from more than one thread. Keep a single writer in mind: this design is for a personal batch cache, not a high-concurrency API.

Step 2: Serve from cache on a hit 

        row = DB.execute('SELECT response FROM cache WHERE key = ?', (cache_key,)).fetchone()
        if row:
            self.send_response(200)
            self.send_header('Content-Type', 'application/json')
            self.end_headers()
            self.wfile.write(row[0].encode())
            return
Enter fullscreen mode Exit fullscreen mode

On a cache hit, the proxy returns the stored JSON immediately. No upstream call, no token spend. The stored value is the raw upstream body, so clients that already speak an OpenAI-compatible chat API do not need a response adapter.

Step 3: Forward to the model on a miss

When the cache misses, the proxy forwards the original body upstream with urllib.request. For the upstream endpoint I used MonkeyCode, an open-source project that provides free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The proxy works with any OpenAI-compatible endpoint, so the upstream is a configuration detail, not a design constraint.

        req = urllib.request.Request(
            UPSTREAM_URL,
            data=json.dumps(body).encode(),
            headers={'Content-Type': 'application/json'},
        )
        with urllib.request.urlopen(req, timeout=60) as resp:
            response = resp.read().decode()
        DB.execute('INSERT INTO cache (key, response) VALUES (?, ?)', (cache_key, response))
        DB.commit()
        self.send_response(200)
        self.send_header('Content-Type', 'application/json')
        self.end_headers()
        self.wfile.write(response.encode())
Enter fullscreen mode Exit fullscreen mode

Define UPSTREAM_URL before you start the server. For example:

UPSTREAM_URL = 'https://api.openai.com/v1/chat/completions'
Enter fullscreen mode Exit fullscreen mode

Or point it at MonkeyCode's endpoint if you are using their free tier. Bind the handler at the bottom of the file:

HTTPServer(('0.0.0.0', 8000), Handler).serve_forever()
Enter fullscreen mode Exit fullscreen mode

That is the entire proxy. Count the lines after you drop comments and you will be under 40.

How to Deploy and Test on a Free Server

The proxy needs an always-on address. I ran it on MonkeyCode's free server option — one Python process, one SQLite file, no database server, no container.

nohup python3 proxy.py &
Enter fullscreen mode Exit fullscreen mode

That is the entire deployment. A single background process on a free server is enough to serve a cache proxy for personal scripts. In your jobs, keep the same request body and change only the base URL to http://your-server:8000.

Send the same request twice and compare wall time:

curl -X POST http://your-server:8000 \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"Summarize the HTTP spec in one sentence"}]}' \
  -w '\nTime: %{time_total}s\n'

curl -X POST http://your-server:8000 \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"Summarize the HTTP spec in one sentence"}]}' \
  -w '\nTime: %{time_total}s\n'
Enter fullscreen mode Exit fullscreen mode

The first call pays model latency. The second should return in milliseconds from SQLite.

Use this checklist before pointing real jobs at the proxy:

  1. Confirm cache.db appears in the working directory after the first request.
  2. Repeat the same curl and confirm time_total drops from seconds to milliseconds.
  3. Run sqlite3 cache.db "SELECT COUNT(*) FROM cache;" — two identical prompts should still be one row.
  4. Change one character in the prompt and confirm a second row appears. That proves the key is exact-match, not fuzzy.

If step 2 does not drop to milliseconds, you are not hitting the same process or the same database file. Check the working directory of nohup and the URL your client actually posts to.

Results, Limitations, and When to Use This Pattern

What I measured

One of my batch jobs generated weekly summaries from a fixed set of inputs. In a single run, it made 214 calls. Only 37 prompts were unique.

The cache turned the other 177 calls into local lookups. That is an 83% reduction in token spend. On a free tier with a fixed allowance, that is the difference between exhausting your quota on day 12 and lasting the full month.

The 83% figure is hit rate on that job, not a universal constant. Your number will track how often the last user message repeats. A weekly report over a stable catalog will look like mine. A chatbot with unique user questions will not.

When to use this pattern

This pattern works best for:

  • Batch jobs — same inputs, repeated runs.
  • Scheduled reports — daily or weekly summaries from fixed data.
  • Retry loops — the same request may be sent multiple times after a timeout.
  • Multi-script workflows — several scripts sharing one cache without sharing memory.

Skip it if your application sends unique prompts every time. A cache proxy then adds latency and complexity for zero benefit.

Limitations and the semantic upgrade path

Exact-match caching has a hard limit: any change to the prompt is a miss. If your prompts include timestamps, user names, or random IDs, the hit rate will be low.

A practical workaround before you jump to embeddings: strip volatile fields from the hashed string. Hash the instruction plus the stable document id, not the full rendered prompt with today's date. That still stays exact-match and still stays under 40 lines.

The upgrade is semantic caching. Embed the prompt, store the vector, and return a cached response when cosine similarity crosses a threshold. That is significantly more code, but the exact-match version is a solid first step and will tell you whether your workload has enough repetition to justify it.

Also, SQLite does not encrypt by default. Do not cache responses containing sensitive data unless you encrypt the database file.

The full proxy is under 40 lines. Point it at any OpenAI-compatible endpoint, run it on a free server, and redirect your scripts to it. If you want to measure your own hit rate first, MonkeyCode's free tier — 10 million tokens and a free server — gives you enough room to run the experiment before committing to a design.

Try it on your next batch job. Log unique keys versus total calls after one run, then share your hit rate in the comments — I would love to see how much you save.

Top comments (0)