A public demo needs a model, but it does not need the credential that pays for it. When free model access and a free server option push the cost of shipping a demo close to zero, the cheapest possible architecture also becomes the riskiest one: paste the provider key into the host's environment variables, forget about it, and hope nobody looks. I want to walk you through a different arrangement, where the key lives in one small process you control and the public server only ever holds a token that expires.
MonkeyCode advertises free model access and a free server option, which is precisely the kind of offer that makes pasting a key into a demo tempting on a Friday afternoon. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am deliberately not quoting token quotas, hardware, model names, or expiry windows here, because those details move faster than blog posts get edited, and a stale number in a security write-up is worse than no number at all; check the project's own documentation before you plan around it.
Two boundaries, not one
Most demos collapse two very different trust relationships into a single process, and that is where the trouble starts. The first boundary runs from the public internet into your demo: anonymous people, crawlers, and anyone who read your post can send whatever they like. The second boundary runs from your demo into the model provider: it carries your credential, your prompt text, and your request volume, and it is the only one of the two that costs you something durable.
When both boundaries live in the same container, a single mistake on the outer one becomes a compromise of the inner one. A verbose stack trace that echoes a header, an unguarded /debug route, or a dependency with a known request-logging bug now hands an attacker a working provider key. Free hosting makes this worse rather than better, because the environment you cannot inspect is exactly the environment where the key sits.
Separate them and the arithmetic changes. The demo can leak everything it knows and the leak is worth a token that dies in ten minutes and can only reach your broker. The broker never sees the internet directly, keeps the provider key in its own process, and answers with text and usage counts instead of upstream headers or tracebacks.
The broker belongs somewhere you control, even if that somewhere is a free server option with a cold start you find annoying. My earlier note that a free host is still a foreign room applies here too: a room you can walk out of is a very different thing from a room where your key lives.
The broker, in about fifty lines
The whole downstream half can be written against the standard library, which matters because every dependency in a credential-holding process is a dependency you have to read. Store the signing secret and the provider key as environment variables that only this process inherits, and never write them to a file inside the repository. The version below assumes Python 3.9 or newer for removeprefix.
# broker.py — holds the provider key, mints short-lived tokens, forwards one endpoint.
import hmac, hashlib, json, os, time, urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
SECRET = bytes.fromhex(os.environ["BROKER_SECRET"]) # 32 random bytes, rotate to revoke all tokens
UPSTREAM = os.environ["MODEL_ENDPOINT"] # your provider's chat endpoint
MODEL_KEY = os.environ["MODEL_KEY"] # never leaves this process
MAX_BYTES = 4000 # hard cap on request body size
def mint(ttl=600):
exp = int(time.time()) + ttl
body = f"v1.{exp}"
sig = hmac.new(SECRET, body.encode(), hashlib.sha256).hexdigest()[:32]
return f"{body}.{sig}"
def valid(tok):
try:
ver, exp, sig = tok.split(".")
except ValueError:
return False
body = f"{ver}.{exp}"
want = hmac.new(SECRET, body.encode(), hashlib.sha256).hexdigest()[:32]
return hmac.compare_digest(want, sig) and int(exp) > time.time()
class Handler(BaseHTTPRequestHandler):
server_version, sys_version = "broker", "" # no version banner in responses
def _send(self, code, payload):
raw = json.dumps(payload).encode()
self.send_response(code)
self.send_header("content-type", "application/json")
self.send_header("content-length", str(len(raw)))
self.end_headers()
self.wfile.write(raw)
def do_POST(self):
if self.path != "/v1/chat":
return self._send(404, {"error": "not_found"})
tok = self.headers.get("authorization", "").removeprefix("Bearer ").strip()
if not valid(tok):
return self._send(401, {"error": "token_rejected"})
n = int(self.headers.get("content-length", 0))
if n > MAX_BYTES:
return self._send(413, {"error": "too_large"})
req = urllib.request.Request(
UPSTREAM,
data=self.rfile.read(n),
headers={"content-type": "application/json",
"authorization": f"Bearer {MODEL_KEY}"},
)
try:
with urllib.request.urlopen(req, timeout=30) as r:
upstream = json.loads(r.read())
except Exception:
return self._send(502, {"error": "upstream_failed"}) # no traceback, no upstream body
text = upstream["choices"][0]["message"]["content"]
return self._send(200, {"text": text, "usage": upstream.get("usage", {})})
if __name__ == "__main__":
ThreadingHTTPServer(("0.0.0.0", 8080), Handler).serve_forever()
Mint tokens from the operator side with python -c 'import broker; print(broker.mint(600))' and hand one to the demo at deploy time. The demo's environment then contains two harmless strings, a URL and a token, and the token itself is the entire blast radius if the container is compromised.
The client half shrinks to something you can read in a sitting, and note what is absent from it: no provider key, no provider hostname, no model routing logic that a visitor could rewrite. Notice also that the demo cannot choose its own model name, which is deliberate, because model selection is a cost and data-handling decision rather than a presentation detail.
const r = await fetch(`${process.env.BROKER_URL}/v1/chat`, {
method: "POST",
headers: { "content-type": "application/json",
authorization: `Bearer ${process.env.SESSION_TOKEN}` },
body: JSON.stringify({ messages: [{ role: "user", content: prompt }] })
});
const { text } = await r.json(); // 401, 413, 502 arrive as short codes, not prose
Proving it holds
A threat model you cannot test is a wish, so here is a probe script you run against your own deployment and nothing else. Each line checks one claim: the client bundle carries no key material, error responses stay terse, unsigned calls fail, oversized bodies fail before they reach the provider, and expired tokens stop working. Run it after every deploy, because a helper library upgrade can silently add a header echo you never wrote.
#!/usr/bin/env bash
# probes.sh — point these two variables at your own demo and broker only.
DEMO="https://your-demo.example"
BROKER="https://your-broker.example"
curl -s "$DEMO/" | grep -Eoi 'sk-[a-z0-9]{8,}|api[_-]?key|model_key' && echo "FAIL: key-shaped string in bundle" || echo "ok: bundle clean"
curl -s -X POST "$DEMO/api/chat" -H 'content-type: application/json' -d '{"prompt":{}}' | grep -Eoi 'traceback|at Object|\.js:[0-9]+' && echo "FAIL: internals leaked" || echo "ok: error terse"
curl -s -o /dev/null -w 'unsigned call -> %{http_code}\n' -X POST "$BROKER/v1/chat" -d '{}'
head -c 20000 /dev/zero | tr '\0' 'a' > big.txt
curl -s -o /dev/null -w 'oversized body -> %{http_code}\n' -X POST "$BROKER/v1/chat" -H "authorization: Bearer $TOKEN" --data-binary @big.txt
for h in server x-powered-by x-aspnet-version; do curl -sI "$BROKER/v1/chat" | grep -i "^$h:" && echo "FAIL: banner $h"; done
Two results deserve a closer look when they appear. A 502 on a malformed prompt is correct and a 200 is not, because the demo should never learn why the provider refused. And if the oversized-body probe returns 401 instead of 413, your size check is sitting behind authentication, which means the token check is doing the work a byte limit should be doing.
The log line for each of these probes is its own small design decision. Record the token expiry, the body size, the verdict, the latency, and a hash of the prompt, then leave the prompt text out entirely; the shape of traffic is what you debug with, and the payload is what you have to protect. That rule has held up well in my own logging notes, and it holds here without modification.
What this costs you, and who should skip it
The broker is not free in the way the hosting is free, so be honest about the tax. You add one network hop of latency, you inherit the cold-start behaviour of whatever free server option you picked, and you now have a second deployment to keep alive. Expiry-based tokens also mean revocation is coarse: rotating BROKER_SECRET kills every live token at once, and per-token revocation needs a deny list you will have to maintain by hand.
Streaming is the other rough edge, and it bites demos that show tokens appearing one at a time. The handler above buffers whole responses, so you would need to pass the upstream stream through without inspecting it, which weakens the guarantee that upstream errors never reach the client. Decide which of those two properties you actually need before you build the feature, because you rarely get both cheaply.
Skip this arrangement in three situations. If your organisation already has a secret manager and a gateway with per-route quotas, you are rebuilding a worse version of something you own. If you handle regulated data or other people's customer records, no shared free infrastructure is appropriate regardless of how clean the broker is, so run the demo locally or do not ship it. And if the demo exists to be read rather than called, a static page with precomputed examples removes the credential problem entirely.
Everything else, which is most weekend demos and internal prototypes, benefits from the split. Start by moving the key out of the demo host today, mint one ten-minute token by hand, and watch the probe script pass before you do anything more ambitious. If you want to try the whole thing on free model access and a free server option, the project's own page is the right place to confirm what is currently included.
Top comments (0)