DEV Community

Dakota Huang
Dakota Huang

Posted on

The Same Prompt Twice Is Double Quota: A Verifiable Cache for Free Model Endpoints

#ai #node #tutorial #llm

Free model endpoints bill quota per token. Send the same prompt twice, and you pay twice. Retries protect you from failures. They do not protect you from duplicates. A cache layer does. This tutorial builds one from zero. Every stage ends with a verification step. No frameworks. No dependencies. One Node.js file.

The problem

CI jobs repeat prompts. Tests re-run the same summarization. Previews regenerate the same completion. Each repeat is a full token bill. Your retry layer handles errors. It cannot handle identical requests. Caching sits in front of the endpoint and answers repeats from memory.

The pattern is small: hash the request, store the response, serve the copy. You get three wins. Lower quota burn. Lower latency. Fewer rate-limit hits.

What you will build

  1. A minimal HTTP forwarder.
  2. An in-memory cache keyed by a SHA-256 hash.
  3. A stats endpoint that reports hits and misses.
  4. A deployment on a free server.
  5. A quota check that proves the savings.

Stage 0 — Prerequisites

You need three things. Node.js 18 or newer. A free model endpoint with an API key. A Linux server you can reach. This walkthrough targets MonkeyCode's free model access and free server option. Both fit the constraints: real endpoints, real rate limits, zero cost. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Stage 1 — A forwarder with a health check

Start with the smallest useful piece. A server that forwards POST bodies to the upstream endpoint. It also exposes /health so you can verify liveness.

// gateway.js — stage 1: forward only
const http = require("http");

const UPSTREAM = process.env.UPSTREAM_URL;
const API_KEY = process.env.UPSTREAM_KEY;
const PORT = process.env.PORT || 8080;

const server = http.createServer(async (req, res) => {
  if (req.method === "GET" && req.url === "/health") {
    res.writeHead(200, { "content-type": "application/json" });
    res.end(JSON.stringify({ ok: true }));
    return;
  }

  const chunks = [];
  for await (const chunk of req) chunks.push(chunk);
  const rawBody = Buffer.concat(chunks).toString("utf8");

  const upstream = await fetch(UPSTREAM, {
    method: "POST",
    headers: {
      "content-type": "application/json",
      authorization: `Bearer ${API_KEY}`,
    },
    body: rawBody,
  });

  const body = await upstream.text();
  res.writeHead(upstream.status, { "content-type": "application/json" });
  res.end(body);
});

server.listen(PORT, () => console.log(`gateway on :${PORT}`));
Enter fullscreen mode Exit fullscreen mode

Run it with your endpoint values.

UPSTREAM_URL="https://your-endpoint.example/v1/chat/completions" \
UPSTREAM_KEY="your-key" \
node gateway.js
Enter fullscreen mode Exit fullscreen mode

Verify in a second terminal.

curl -s http://localhost:8080/health
# {"ok":true}

curl -s -X POST http://localhost:8080/ \
  -H "content-type: application/json" \
  -d '{"prompt":"explain TCP backoff in one sentence"}' \
  -o /dev/null -w "%{http_code}\n"
# 200
Enter fullscreen mode Exit fullscreen mode

The health check proves the process is alive. The POST proves the upstream path works. Stage 1 is done.

Stage 2 — Add the cache

The cache key is a SHA-256 hash of method, URL, and raw body. Same body, same key. Different body, different key. No JSON parsing. No schema guessing.

Add these lines.

const crypto = require("crypto");

const TTL_MS = Number(process.env.CACHE_TTL_MS || 60_000);
const cache = new Map();
let hits = 0;
let misses = 0;

function cacheKey(method, url, rawBody) {
  return crypto
    .createHash("sha256")
    .update(`${method} ${url} ${rawBody}`)
    .digest("hex");
}
Enter fullscreen mode Exit fullscreen mode

Then replace the upstream call with a cache check.

const key = cacheKey(req.method, req.url, rawBody);
const cached = cache.get(key);

if (cached && cached.expiresAt > Date.now()) {
  hits++;
  res.writeHead(cached.status, {
    "content-type": "application/json",
    "x-cache": "HIT",
  });
  res.end(cached.body);
  return;
}

misses++;
const upstream = await fetch(UPSTREAM, {
  method: "POST",
  headers: {
    "content-type": "application/json",
    authorization: `Bearer ${API_KEY}`,
  },
  body: rawBody,
});
const body = await upstream.text();

if (upstream.status >= 200 && upstream.status < 300) {
  cache.set(key, {
    status: upstream.status,
    body,
    expiresAt: Date.now() + TTL_MS,
  });
}

res.writeHead(upstream.status, {
  "content-type": "application/json",
  "x-cache": "MISS",
});
res.end(body);
Enter fullscreen mode Exit fullscreen mode

Two details matter. Only 2xx responses enter the cache. Error responses pass through untouched. The TTL expires entries after 60 seconds by default. You can change it with CACHE_TTL_MS.

Restart the server. Verify the cache with the same request twice.

curl -s -X POST http://localhost:8080/ \
  -H "content-type: application/json" \
  -d '{"prompt":"explain TCP backoff in one sentence"}' \
  -D - -o /dev/null | grep -i x-cache
# x-cache: MISS

curl -s -X POST http://localhost:8080/ \
  -H "content-type: application/json" \
  -d '{"prompt":"explain TCP backoff in one sentence"}' \
  -D - -o /dev/null | grep -i x-cache
# x-cache: HIT
Enter fullscreen mode Exit fullscreen mode

The header is your proof. MISS means one upstream call. HIT means zero.

Stage 3 — Add the stats endpoint

A cache without numbers is a belief. Add /stats to report hits, misses, and hit ratio.

if (req.method === "GET" && req.url === "/stats") {
  const total = hits + misses;
  res.writeHead(200, { "content-type": "application/json" });
  res.end(JSON.stringify({
    hits,
    misses,
    hitRatio: total ? Number((hits / total).toFixed(3)) : 0,
    cacheSize: cache.size,
  }));
  return;
}
Enter fullscreen mode Exit fullscreen mode

Restart and verify.

curl -s http://localhost:8080/stats
# {"hits":1,"misses":1,"hitRatio":0.5,"cacheSize":1}
Enter fullscreen mode Exit fullscreen mode

The ratio tells you if caching is worth it for your workload. 0.5 means half the requests never reached the model.

Stage 4 — The complete file

Here is the full gateway. Save it as gateway.js.

// gateway.js — cache-first proxy for free model endpoints
const http = require("http");
const crypto = require("crypto");

const UPSTREAM = process.env.UPSTREAM_URL;
const API_KEY = process.env.UPSTREAM_KEY;
const PORT = process.env.PORT || 8080;
const TTL_MS = Number(process.env.CACHE_TTL_MS || 60_000);

const cache = new Map();
let hits = 0;
let misses = 0;

function cacheKey(method, url, rawBody) {
  return crypto
    .createHash("sha256")
    .update(`${method} ${url} ${rawBody}`)
    .digest("hex");
}

async function callUpstream(rawBody) {
  const res = await fetch(UPSTREAM, {
    method: "POST",
    headers: {
      "content-type": "application/json",
      authorization: `Bearer ${API_KEY}`,
    },
    body: rawBody,
  });
  const body = await res.text();
  return { status: res.status, body };
}

const server = http.createServer(async (req, res) => {
  if (req.method === "GET" && req.url === "/health") {
    res.writeHead(200, { "content-type": "application/json" });
    res.end(JSON.stringify({ ok: true, cacheSize: cache.size }));
    return;
  }

  if (req.method === "GET" && req.url === "/stats") {
    const total = hits + misses;
    res.writeHead(200, { "content-type": "application/json" });
    res.end(JSON.stringify({
      hits,
      misses,
      hitRatio: total ? Number((hits / total).toFixed(3)) : 0,
      cacheSize: cache.size,
    }));
    return;
  }

  if (req.method !== "POST") {
    res.writeHead(405, { "content-type": "application/json" });
    res.end(JSON.stringify({ error: "method not allowed" }));
    return;
  }

  const chunks = [];
  for await (const chunk of req) chunks.push(chunk);
  const rawBody = Buffer.concat(chunks).toString("utf8");

  const key = cacheKey(req.method, req.url, rawBody);
  const cached = cache.get(key);

  if (cached && cached.expiresAt > Date.now()) {
    hits++;
    res.writeHead(cached.status, {
      "content-type": "application/json",
      "x-cache": "HIT",
    });
    res.end(cached.body);
    return;
  }

  misses++;
  const upstream = await callUpstream(rawBody);

  if (upstream.status >= 200 && upstream.status < 300) {
    cache.set(key, {
      status: upstream.status,
      body: upstream.body,
      expiresAt: Date.now() + TTL_MS,
    });
  }

  res.writeHead(upstream.status, {
    "content-type": "application/json",
    "x-cache": "MISS",
  });
  res.end(upstream.body);
});

server.listen(PORT, () => console.log(`gateway on :${PORT}`));
Enter fullscreen mode Exit fullscreen mode

The cache is a plain Map. It grows until entries expire. That is fine for a single-user gateway. For a busy proxy, add a size cap or a periodic sweep.

Stage 5 — Deploy to a free server

Copy the file to your server. The commands assume SSH. If your free server exposes a web terminal instead, run the same commands inside that shell.

scp gateway.js user@your-server:/opt/gateway/
Enter fullscreen mode Exit fullscreen mode

Create a systemd unit so the gateway survives reboots.

[Unit]
Description=model cache gateway
After=network-online.target

[Service]
Environment=UPSTREAM_URL=https://your-endpoint.example/v1/chat/completions
Environment=UPSTREAM_KEY=your-key
Environment=PORT=8080
ExecStart=/usr/bin/node /opt/gateway/gateway.js
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

Install and start it.

sudo systemctl daemon-reload
sudo systemctl enable --now gateway
Enter fullscreen mode Exit fullscreen mode

Verify remotely.

curl -s http://your-server:8080/health
# {"ok":true,"cacheSize":0}
Enter fullscreen mode Exit fullscreen mode

If the request times out, check the provider's firewall rules. Some free servers block inbound ports by default. Allow 8080 in the provider dashboard.

Then repeat the cache check against the remote address.

curl -s -X POST http://your-server:8080/ \
  -H "content-type: application/json" \
  -d '{"prompt":"explain TCP backoff in one sentence"}' \
  -D - -o /dev/null | grep -i x-cache
# x-cache: MISS

curl -s -X POST http://your-server:8080/ \
  -H "content-type: application/json" \
  -d '{"prompt":"explain TCP backoff in one sentence"}' \
  -D - -o /dev/null | grep -i x-cache
# x-cache: HIT
Enter fullscreen mode Exit fullscreen mode

Stage 5 is done. The gateway runs outside localhost.

Stage 6 — Prove the savings

Run ten identical requests inside the TTL window.

for i in $(seq 1 10); do
  curl -s -X POST http://your-server:8080/ \
    -H "content-type: application/json" \
    -d '{"prompt":"explain TCP backoff in one sentence"}' \
    -o /dev/null
done

curl -s http://your-server:8080/stats
# {"hits":10,"misses":1,"hitRatio":0.909,"cacheSize":1}
Enter fullscreen mode Exit fullscreen mode

Ten requests. One upstream call. Nine saved. Now open the model provider's usage dashboard. The token count for that prompt should show one charge, not ten. That dashboard is the external proof. The stats endpoint is the internal proof.

Limitations

The cache only helps repeated prompts. Unique prompts always miss. The cache lives in memory. A restart clears it, and the first repeat burns quota again. The TTL serves stale text. If the model output drifts, you serve old completions until expiry. The key is the raw body. Different whitespace means a different key. Normalize your JSON if that matters. This is not a retry layer. A 429 or 5xx passes through as-is. Pair it with a retry layer if you need both. The gateway buffers full responses. Streaming requests will not behave well.

Who should not use this

Skip this pattern for chat workloads with unique user messages. The hit ratio will sit near zero. Skip it for streaming completions. The buffer defeats the purpose. Skip it for personalized outputs. Serving user A's cached response to user B is a correctness bug. Skip it for high-throughput production. You want Redis or a CDN, not a Map in one Node process.

The takeaway

The pattern is small: hash, store, check, serve. It saves quota on every repeated prompt. It turns a rate-limit headache into a cache hit. Start with the full file above. Measure your hit ratio for a week. If it stays above 0.2, the cache is paying for itself. If you need a free endpoint to test against, MonkeyCode's free model access is a reasonable place to start.

Top comments (0)