A Cache-First LLM Gateway You Can Run on a Free Server

You have four API keys, three SDKs, and no idea which prompt hit which model last week. Each script calls the provider directly, pays its own token bill, and forgets every response. When the bill arrives, nobody can say why. The fix is not another dashboard. It is a small gateway that sits between your code and the LLM providers. It caches, logs, and routes requests through one endpoint.

A gateway does not need a big machine. It needs to stay awake, hold a little disk, and forward JSON. A free server fits this perfectly. MonkeyCode offers free models and a free server, so you can build the whole thing without opening your wallet. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free models can serve as the downstream model when you want zero-cost experiments. The free server gives your gateway a permanent home and a public URL.

The Cache-First Pattern

Most LLM calls are repeats. The same test input, the same error message, the same support question. Why spend money or tokens twice? A cache-first gateway stores every response in a local store, keyed by a hash of the request. When the same request arrives again, it returns the stored answer without calling a model. This cuts latency, saves tokens, and makes your eval runs deterministic.

You still need a fallback for genuinely new requests. The gateway forwards them to a configured LLM endpoint, then stores the fresh response. That downstream model can be a paid provider or MonkeyCode's free models. Your application code does not care which one you use.

The Code

The gateway is a single FastAPI file. It exposes a /v1/chat/completions route so OpenAI SDKs can point at it without changes. Here is the complete implementation:

import hashlib
import json
import os
import sqlite3
import time
from typing import Any, Dict, Optional

import httpx
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

app = FastAPI()
DB_FILE = os.environ.get("CACHE_DB", "gateway.db")
DOWNSTREAM_URL = os.environ.get("DOWNSTREAM_URL")
DOWNSTREAM_KEY = os.environ.get("DOWNSTREAM_API_KEY")
DEFAULT_MODEL = os.environ.get("DEFAULT_MODEL", "default")

conn = sqlite3.connect(DB_FILE, check_same_thread=False)
conn.execute("""
    CREATE TABLE IF NOT EXISTS cache (
        key TEXT PRIMARY KEY,
        response TEXT,
        created_at REAL
    )
""")
conn.commit()

class ChatRequest(BaseModel):
    model: Optional[str] = None
    messages: list[dict[str, Any]]
    temperature: Optional[float] = None
    max_tokens: Optional[int] = None

class ChatResponse(BaseModel):
    id: str
    object: str = "chat.completion"
    created: int
    model: str
    choices: list[dict[str, Any]]
    usage: dict[str, int]

@app.post("/v1/chat/completions")
async def chat(req: ChatRequest):
    payload = req.dict(exclude_none=True)
    payload["model"] = req.model or DEFAULT_MODEL

    key = hashlib.sha256(
        json.dumps(payload, sort_keys=True).encode()
    ).hexdigest()

    cached = get_cached(key)
    if cached is not None:
        return ChatResponse(**cached)

    if not DOWNSTREAM_URL:
        raise HTTPException(500, "DOWNSTREAM_URL not set")

    headers = {"Content-Type": "application/json"}
    if DOWNSTREAM_KEY:
        headers["Authorization"] = f"Bearer {DOWNSTREAM_KEY}"

    async with httpx.AsyncClient() as client:
        r = await client.post(
            DOWNSTREAM_URL,
            json=payload,
            headers=headers,
            timeout=60.0,
        )
        r.raise_for_status()
        data = r.json()

    set_cached(key, data)
    log_request(payload, data)
    return ChatResponse(**data)

def get_cached(key: str) -> Optional[dict]:
    row = conn.execute(
        "SELECT response FROM cache WHERE key = ?", (key,)
    ).fetchone()
    if row is None:
        return None
    return json.loads(row[0])

def set_cached(key: str, data: dict):
    conn.execute(
        "INSERT OR REPLACE INTO cache VALUES (?, ?, ?)",
        (key, json.dumps(data), time.time()),
    )
    conn.commit()

def log_request(payload: dict, response: dict):
    with open("access.log", "a") as f:
        f.write(json.dumps({"ts": time.time(), "req": payload, "resp": response}) + "\n")

The cache lives in SQLite. A log file records every miss and every provider response. You can inspect both later to understand usage and debug prompts. No external services needed.

How to Route to Free Models

Set DOWNSTREAM_URL to the endpoint that MonkeyCode exposes for its free models. Your provider documentation will show the exact path. The gateway does not care about the provider name. It only forwards JSON. You can also switch between free and paid models by changing one environment variable and restarting the service.

For local testing, point DOWNSTREAM_URL to any OpenAI-compatible mock. This makes the gateway testable before you connect real credentials.

Deploying on the Free Server

MonkeyCode's free server gives you a Linux environment with a public URL. Your workflow looks like this:

1. SSH into the server or clone the repo in its web terminal.
2. Create a virtualenv and install dependencies: pip install fastapi uvicorn httpx pydantic.
3. Set the environment variables.
4. Run uvicorn main:app --host 0.0.0.0 --port 8000.

Add a simple process manager so it survives crashes. The free server typically supports systemd or at least a cron-based restart script. A minimal supervisor loop in Python works too.

Once the gateway is live, update your applications to point at https://your-free-server-url/v1/chat/completions. All requests now flow through cache and log. Your keys stay on the gateway. You can even revoke direct access to your paid provider and force everyone through the gateway.

When to Use This Pattern

Situation	Recommendation
A handful of scripts calling OpenAI directly	Use the gateway immediately
Hundreds of concurrent users on a public app	Not for this free server
You need deterministic replay for eval suites	Cache-first is a perfect fit
You cannot tolerate stale answers	Set a TTL or bypass on demand

Limitations

This gateway is not a production load balancer. It has no retry logic, no circuit breaker, and no multi-tenant auth. The cache grows forever unless you prune old rows. The free server may be a shared tenant, so CPU spikes from another user can slow your response times.

More importantly, free models are not guaranteed to be available forever. They may change their rate limits, latency, or filtering. Your gateway should treat free models as a convenience, not a contract. Keep the downstream endpoint configurable so you can migrate the moment you need stability.

Also, cache keys are strict. A single character change in the system prompt produces a full cache miss. If your prompts are highly dynamic, expect a low hit rate. In that case, add semantic caching later; the current version is deliberately simple.

Who Should Skip This

If you only write one script and never touch it again, a gateway is overkill. If you need guaranteed uptime and low p99 latency, buy a managed service. If you cannot afford a 60-second timeout on any request, rethink your architecture.

Otherwise, this tiny gateway will save you tokens, cut your latency on repeat work, and give you an honest log of what you actually send to models. Start with one environment variable and one route. You can always grow it later.

Try it this week. Point one throwaway script at it and watch the cache hit count go up. That single change teaches you more about your own usage than any dashboard will.