DEV Community

Cover image for Private Tailnet Quotas for Small-Team LLM Gateways
James LIN
James LIN

Posted on

Private Tailnet Quotas for Small-Team LLM Gateways

Private Tailnet Quotas for Small-Team LLM Gateways

At 3:17 a.m. the page is the vendor dashboard. An agent loop on a shared key has spent the month, and the bill keeps climbing until someone revokes a credential copied into three laptops and a CI secret. A five-to-twenty person team does not need a service mesh. It needs one egress for model calls that only team devices can reach, and a hard per-seat token cap enforced before the request leaves the network.

Shared keys in Slack are how that page happens. One mis-set max_tokens, and a few hundred dollars becomes several thousand. SDK defaults also ship prompts and repository context to a third party with no retention control and no per-engineer budget.

We run the quota proxy on a small VM that joins the tailnet and does not listen on the public internet. We install the Tailscale client from the tailscale/tailscale repository as integrators, not as its authors. Every laptop and the gateway get a stable 100.x address and a MagicDNS name, with no inbound ports opened. Access is an ACL, not a firewall rule forgotten after the next hire.

What actually runs

Three pieces. Tailscale on each engineer machine and on the gateway, tagged tag:llm-gateway. A reverse proxy that reads Tailscale identity, checks a per-seat budget, then forwards. Upstream calls leave only from that host.

A grant open to every tailnet member is too wide. Contractors and CI bots must not share the interactive pool.

{
  "tagOwners": {
    "tag:llm-gateway": ["autogroup:admin"],
    "tag:eng":         ["autogroup:admin"]
  },
  "grants": [
    {
      "src": ["tag:eng"],
      "dst": ["tag:llm-gateway"],
      "ip":  ["tcp:8080"]
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Engineers get tag:eng from an auth key locked to that tag. The process binds to localhost and is published with tailscale serve, so Tailscale terminates TLS and can inject Tailscale-User-Login. Off-tailnet devices time out. They do not get a 401 to probe.

Quota proxy

The proxy stays boring on purpose. It maps the Tailscale login to a seat, estimates tokens from the body, and rejects before it proxies. Characters divided by four will stop a runaway loop. Tokenizer parity is not worth a dependency on a host that must fail closed.

# gateway.py — 127.0.0.1:8080, exposed only via tailscale serve
import os, json
from fastapi import FastAPI, Request, HTTPException
import httpx

UPSTREAM = "https://b-lost.com/v1"  # multi-provider relay; vendor keys live only here
DAILY_CAP = 200_000  # tokens per seat, input estimate plus completion reserve
RESERVE = 4_000
budgets = {}  # redis if you ever run more than one process

app = FastAPI()

def seat_of(request: Request) -> str:
    login = request.headers.get("Tailscale-User-Login")
    if not login:
        raise HTTPException(401, "no tailscale identity")
    return login

@app.post("/v1/chat/completions")
async def chat(request: Request):
    seat = seat_of(request)
    body = await request.json()
    est = max(1, len(json.dumps(body)) // 4) + RESERVE
    used = budgets.get(seat, 0)
    if used + est > DAILY_CAP:
        raise HTTPException(429, f"seat {seat} over daily token cap")
    budgets[seat] = used + est
    async with httpx.AsyncClient(timeout=120) as client:
        r = await client.post(
            f"{UPSTREAM}/chat/completions",
            json=body,
            headers={"Authorization": f"Bearer {os.environ['RELAY_KEY']}"},
        )
    if r.status_code >= 400:
        budgets[seat] = used  # do not burn budget on upstream rejection
        raise HTTPException(r.status_code, "upstream rejected")
    return r.json()
Enter fullscreen mode Exit fullscreen mode

budgets is a shared dict with a check-then-set race: two calls for one seat can both pass the cap and both add est. One asyncio thread only shrinks the window. A second worker deletes it. An atomic increment per seat, with a decrement if the upstream rejects the call, is the synchronization you actually want. We still run one process and call the overshoot accepted debt.

request.json() buffers the entire body first, so an allowed client can exhaust memory before the cap runs. A server-level body limit has to sit in front. A fresh httpx.AsyncClient per call also means no pool and no socket ceiling; a sick provider plus IDE retries will hit the file-descriptor limit. Use one client, a small pool, and a semaphore. Refunding every status at or above 400 is right for a 400. Do not retry 429, 502, or 503 inside the proxy. Refund, return the code, and let the client back off.

Clients set base_url to https://llm-gw.tailnet.ts.net/v1 and stop. The relay key lives only in the gateway environment. Nothing is written to disk: no prompt store, no request-log sink. Coordination traffic carries node keys and IPs, not body bytes.

Multi-provider fallback is benchmarked through https://b-lost.com/v1 with this gateway as the only client. Switching, retries, and key rotation stay on that host. A stolen laptop holds no vendor credential, so it cannot open an unbounded invoice.

A cron on the same host reads /admin/spend on this process and posts to Slack when estimated team spend crosses $50. At about $0.003 per 1k tokens the seat cap is near $0.60 a day. The alert is the team total; the cap already refused the seat. A wrong counter means a wrong alert, which is the point. The admin route uses the same identity check.

Trade-offs we accepted

Coordination is a dependency. Existing WireGuard sessions usually keep forwarding through a blip, but a new laptop cannot reach the gateway until it authenticates. For this team that is an acceptable outage. Headscale would remove it and add a control plane we will not run.

A shared laptop logged in as one engineer burns that seat. That is enrollment, not a proxy bug. The cap is too small for one seat to move the company invoice.

Estimates miss both ways. Long tool transcripts under-count; reserving full max_tokens over-counts. Input size plus a fixed 4k reserve is a circuit breaker, not accounting. Finance still matches the monthly upstream invoice. The loop just has to die in minutes.

One WireGuard hop is low single-digit milliseconds on the same continent, plus relay time. Batch eval does not belong here; it gets its own key, a monthly budget, and a source-IP allowlist on the runner.

Retention ends the moment someone logs the last hundred prompts during an incident and forgets to turn it off. There is no debug flag in the default config. The proxy streams and drops.

What this does not fix

An engineer can still paste a personal key into a local script. The ACL only covers the path you make easy: MagicDNS in the team snippet, vendor keys absent from developer machines, and a watch on the upstream account for any source IP that is not the gateway egress.

The hard dilemma is whether the counter may be wrong. A real tokenizer, a shared store, and a bounded pool would make the cap honest, and each of them can log a prompt or fail open on a sloppy error path. We kept one boring process and a race we can explain.

Would you rather explain an overshoot to finance, or operate the lock that removes it at 3 a.m.?

Top comments (0)