You stood up vLLM on a box with a GPU. It works. You share the URL with a couple of teams and go back to your actual job.
Here is what you just deployed:
No authentication. vLLM doesn't ship with any by default. Anyone who learns the address can use it — including whatever is scanning your internal network.
No limits. One colleague's batch evaluation script forgets a sleep, sends two million requests overnight, and the GPU queue is saturated by morning. The customer-support bot that shares the box starts timing out.
No record. Someone asks whether customer data has been sent to the model. You cannot answer. The access log has IP addresses and paths.
No content controls. The model happily repeats a phone number out of its RAG context to whoever asked.
None of this is vLLM's fault — it's an inference engine, not a gateway. But somebody has to put the door on the building.
InferGuard is that door: a small Go reverse proxy that sits in front of any OpenAI-compatible backend and adds virtual API keys, per-key rate limiting, audit logging, and bidirectional PII redaction. Applications don't change — you point base_url at the gateway and everything else stays the same.
Sixty seconds to running
You don't need a real model to try it. The repo ships a mock backend that streams a response with PII deliberately split across chunks.
git clone https://github.com/songzhouran/InferGuard && cd InferGuard
cp config.example.yaml config.yaml
# point the gateway at the mock backend instead of Ollama
sed -i 's|localhost:11434|localhost:9901|' config.yaml
go run ./hack/mockllm & # fake backend on :9901
go run ./cmd/inferguard -config config.yaml # gateway on :8080
Now call it the way you'd call OpenAI:
curl -sN http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer ig-demo-change-me" \
-H "Content-Type: application/json" \
-d '{"model":"llama3","stream":true,"messages":[{"role":"user","content":"hi"}]}'
The mock backend streams The contact's phone is 138 / 1234 / 5678, email a / lice@corp.example.com... as four separate SSE events, 80 ms apart. What arrives at your terminal:
data: {"choices":[{"delta":{"content":""},"index":0}],"id":"cmpl-mock",...}
data: {"choices":[{"delta":{"content":""},"index":0}],"id":"cmpl-mock",...}
data: {"choices":[{"delta":{"content":""},"index":0}],"id":"cmpl-mock",...}
data: {"choices":[{"delta":{"content":"The contact's ph"},"index":0}],...}
data: {"choices":[{"delta":{"content":"one is [CN_PHONE_1], email [EMAIL_1] for the record."},"index":0}],...}
data: [DONE]
The phone number and the email are gone, even though neither was ever complete inside a single event.
You can also see the cost of the mechanism in that output. The gateway holds back a fixed 64-byte tail, and this entire demo response is only about 90 bytes — so the first three events carry no text at all, and everything lands near the end. On a real response of a few hundred tokens, that same fixed 64 bytes is a constant lag of a few dozen milliseconds and you won't perceive it. The demo is deliberately the worst case for the mechanism, because it's the best case for showing what the mechanism is for.
(The release point is a byte boundary, not a word boundary — hence The contact's ph / one is …. Clients concatenate deltas, so this is invisible in any real UI.)
What it actually does
Virtual API keys
Clients get a gateway key. The real upstream credential lives only in the gateway config and is swapped in on the way out.
upstream:
base_url: "http://localhost:8000/v1"
api_key: "${UPSTREAM_API_KEY}"
keys:
- {key: "ig-algo-x1", name: "algo", rpm: 2000, tpm: 500000}
- {key: "ig-cs-y2", name: "cs", rpm: 600, tpm: 100000}
- {key: "ig-admin-z3", name: "admin", rpm: 60, tpm: 20000}
Someone leaves the team, you delete a line. The backend never changes, and the real key was never on anyone's laptop.
Rate limits that isolate blast radius
Token buckets per key, on both requests-per-minute and estimated tokens-per-minute. The point isn't stinginess — it's that the runaway script hits its own ceiling and gets a 429 at the gateway, so the support bot on the same GPU never notices.
Two details that took a round of fixing: completion tokens for streams are counted from the actual text content rather than the SSE wire envelope (counting the envelope over-charges by 15–50×), and a request rejected by the token bucket doesn't burn request-budget on its way out.
Audit log
One JSON line per request. Metadata only — never prompt or completion content, so the audit file doesn't become the very thing you're trying to protect.
{"ts":"2026-08-15T02:37:37Z","request_id":"2d9f2074","key":"demo",
"path":"/v1/chat/completions","model":"llama3","stream":false,"status":200,
"latency_ms":1,"prompt_tokens_est":28,"completion_tokens_est":39,
"pii_input":{"CN_ID":1,"CODENAME":1},"pii_output":{"CN_PHONE":1,"CREDIT_CARD":1}}
PII findings are recorded as per-category counts. Every request produces a line, including upstream failures — a 502 when your backend is down is exactly when you want the record to exist.
jq 'select(.blocked)' audit.jsonl # who's getting throttled
jq -s 'group_by(.key)|map({key:.[0].key,
n:map(.completion_tokens_est)|add})' audit.jsonl # usage by team
PII redaction, both directions
Inbound, so an employee pasting a customer list into a prompt doesn't ship it to a third-party model. Outbound, so a model that picked up real records from RAG or fine-tuning doesn't hand them to whoever asked.
Built-in detectors: China resident ID, China mobile, UnionPay, Visa/Mastercard/Amex, US SSN, email. Plus your own patterns:
pii:
enabled: true
input: redact # redact | block | alert | off
output: redact
mode: label # [CN_PHONE_1] — or "mask" for 138****5678
custom:
- name: project-codename
regex: "(?i)Project\\s+Phoenix"
category: CODENAME
The part I'd point at is precision. \b\d{18}\b matches a Chinese ID card — and also order numbers, hashes, and concatenated timestamps. So every regex hit must additionally pass a checksum: GB 11643 for ID cards, Luhn for bank cards, structural rules for SSNs. A made-up 18-digit number streams through untouched; a real one gets caught. That difference is what decides whether a redaction tool stays switched on.
label mode assigns a stable tag per distinct value, so [CN_PHONE_1] refers to the same person throughout a response and the text stays coherent for the model and the reader.
The hard part is streaming
Everything above is ordinary proxy work. The interesting problem is that an LLM emits tokens, and a token boundary has no relationship to a semantic boundary. A phone number arrives as 138 + 1234 + 5678. Scan any single chunk and you find nothing.
Buffering the whole response before scanning is correct and destroys the streaming UX. Scanning each chunk independently keeps the UX and doesn't work.
InferGuard holds back a fixed-size tail — 64 bytes by default — of every text field, then scans across the boundary. Any entity shorter than that window is guaranteed to be seen whole no matter where chunk boundaries fall, in any script, for any pattern you configure. What's held when the stream ends is flushed through a final scan as a synthetic delta, so trailing PII is neither lost nor leaked.
I originally wrote something cleverer — a heuristic that only held back bytes that "looked like" the start of an entity — and it leaked three different ways. That story, with the code, is in a separate post.
Fail closed
A proxy's instinct with content it can't parse is to pass it through. For a redaction gateway that instinct is a bypass: add Content-Encoding: gzip and the control silently stops applying.
So InferGuard refuses instead. Gzip request bodies are decompressed and scanned; other encodings get a 415. An SSE stream carrying an event that won't parse is terminated, not forwarded. An oversized response is a 502, not a truncated 200. Every refusal lands in the audit log with a reason.
Deploying it
docker build -t inferguard .
docker run -p 8080:8080 -v $PWD/config.yaml:/etc/inferguard/config.yaml inferguard
Distroless, non-root, single static binary, no dependencies beyond a YAML parser. GET /healthz for your load balancer. The gateway is stateless apart from in-memory rate-limit buckets, so it scales horizontally — with the caveat that limits currently apply per instance.
It sends one anonymous ping at startup (random instance ID, version, OS/arch — no prompts, no config, no keys). telemetry: false or INFERGUARD_TELEMETRY=off turns it off.
What it doesn't do yet
Being explicit, because a security tool that oversells its coverage is worse than none:
Prompt-injection detection is on the roadmap, not in the build. The rule engine is the next piece of work.
Entities longer than max_hold bytes fall outside the streaming guarantee.
Rate limits are in-process, so a multi-instance deployment multiplies the effective limit. Redis-backed limiting is planned.
Token counts are estimated at ~4 characters per token, not properly tokenized.
Only the OpenAI-compatible protocol is supported today. Anthropic's Messages API uses a different SSE shape and needs its own codec — the redaction engine itself is transport-agnostic, so it's a codec, not a rewrite.
If you want to poke at it
The design seam I care most about is that the sliding window takes and returns plain strings and knows nothing about SSE, so a model-based detector or a different wire protocol slots in behind the same interface. That's where the interesting contributions are.
Apache-2.0, Go, ~1500 lines: github.com/songzhouran/InferGuard
If you run self-hosted inference, I'd genuinely like to know what your gateway layer looks like today — I suspect the honest answer for a lot of teams is "nothing," which is the reason I built this.
Top comments (0)