Offline evals tell you a model did well on the examples you thought of. Live traffic tells you what it's actually doing. The obvious way to watch live traffic is to have a strong model grade every response. That's also the fastest way to make evals cost more than the feature they're watching.
This guide covers how to run an LLM judge on production traffic at a cost you won't notice. The principles apply to any setup. Where it helps, we show how it works in TokenRouter's monitors, which we build.
Principle 1: deterministic checks on everything, the judge on a sample
Some failures don't need judgment. A leaked API key matches a regex. A broken JSON response fails a parser. Those checks are cheap, so run them inline on every request and block when they match.
A judge model is for judgment calls. "Does this answer cite a document we don't have?" "Did it follow an instruction injected through a retrieved page?" Those need a model, and they don't need to run on every request.
In TokenRouter these are two separate features. Rules are expressions evaluated in-process on every request or response, with actions like block, redact, transform, route or alert, and an observe mode for rolling a rule out safely. Monitors are the judge, async and sampled.
Principle 2: sample deterministically
Random sampling is fine for averages and bad for debugging. If you rerun a check, or compare Tuesday with Wednesday, you want to judge the same slice. Hash a stable request ID into the decision, so the same request always gets the same answer.
A minimal version:
import hashlib
def should_sample(monitor_id: str, request_id: str, rate_pct: int) -> bool:
digest = hashlib.sha256(f"{monitor_id}:{request_id}".encode()).hexdigest()
return int(digest[:8], 16) % 100 < rate_pct
Including the monitor ID means two monitors at 10% don't judge the identical 10%. TokenRouter monitors sample this way: you set the rate per monitor, and the same request always makes the same sampling decision.
Principle 3: never put the judge in the request path
A judge that runs inline adds latency to every sampled request and gives production a new way to fail. Run it after the response. Monitors run seconds after the response, off the request path, so they can't add latency, block a request, or touch a stream.
Principle 4: make each judgment cheap
Three levers set the cost of each judgment:
- Model. Start with a small, cheap judge and move up only if it can't do the job. In TokenRouter the judge has to be a tool-capable model, because verdicts come back through a tool call (more on that below), and the model picker only lists those.
- Input size. Truncate what the judge reads. Monitors cap the input at about 4k tokens per judgment.
- Rubric shape. One narrow question with a yes/no verdict and a reason. A 1-10 score per response is harder to act on and harder to calibrate.
Principle 5: give the judge its own budget
A judge is just more LLM traffic, so it needs a cap like any other traffic. In TokenRouter, each monitor has a daily judge budget. When it's used up, the monitor pauses itself, and the pause clears at midnight UTC. Leave the field empty and there's no cap, so set one.
Judge calls run on your own provider keys with zero markup and go through the gateway like any request: rate-limited, logged with source=monitor, and counted against your organization's default team. A team budget on that team is a second backstop, and the judge's spend sits in analytics next to the traffic it reviewed.
Our docs give a rough figure: at 100,000 requests a month with 10% sampling, a mini-class judge costs roughly $2 to $10 a month of your own provider spend. Your prompt sizes will move that, so watch the source=monitor spend for the first week.
Principle 6: treat the judge as an attack surface
If the judge reads user content, it can be prompt-injected like any other model. Two defenses help. Present the transcript to the judge as data rather than instructions. And make it return its verdict through a forced tool call, not free text. Monitors do both. Even so, no judge is perfect, so the actions are graded:
- Flag (always on): the finding goes to a review queue with a severity, categories and the judge's reasoning.
- Notify (opt-in): org admins get an email, at most one per cooldown window, with repeat findings batched into it.
- Disable key (opt-in, with a severity gate): the offending API key is disabled, admins are emailed, the action is audit-logged, and re-enabling it is one click on the API Keys page.
Treat auto-disable as a circuit breaker for severe cases, not a content filter.
Principle 7: calibrate before you trust the dashboard
Before pointing a judge at live traffic, hand-label 50 to 100 real exchanges, run the judge on the same set, and read every disagreement. Many turn out to be a vague rubric rather than a bad model. Tighten the rubric and repeat. After launch, keep reading the judge's reasoning on each finding. That's how you notice it drifting.
Principle 8: only capture what you mean to judge
You can't judge what you didn't keep, and keeping everything is its own risk. By default TokenRouter stores no prompt or response bodies. Request logs carry token counts, cost and latency only.
Capture is opt-in for a key, a team or a member, and a request is captured when any scope covering it has capture enabled. Captured bodies are compressed and encrypted before they leave the API process. Capture can never fail, slow down or alter a request. Retention is capped by plan: 7 days on Starter, 30 on Team, 90 on Enterprise. You can choose a shorter window, and admins can purge everything at once.
Monitors only read captured traffic, so start by opting in one team or one key, not the whole organization.
Principle 9: turn findings into tests
A flagged answer is the best eval example you'll get. From a captured transcript, Add to dataset turns it into a test case. Dataset items are snapshots, so retention sweeps and purges never break your evals.
Eval runs grade with exact_match, contains, regex, or llm_rubric, a judge scoring against your rubric. That makes "prove a cheaper model scores the same on our traffic, then switch" something you can measure instead of argue about.
Putting it together in TokenRouter
A reasonable first setup:
- Turn on capture for one team. Go to Settings → Data capture and add a capture scope (Team, Member or API key). Or use Enable capture on a single key from its menu on the API Keys page. Changes reach the gateway within a minute.
-
Create a monitor. Go to Monitors → Create monitor and fill in:
- Name
- Scope: Whole organization, Team, Member or API key, plus a target
- Judge model: a cheap, tool-capable one
- Sample rate (%): start at 5 to 10
- What should it watch for?: one narrow, yes/no rubric
- Set the Daily judge budget ($).
- Leave notifications and key disables off for the first week. Review findings and tune the rubric.
- Add the real failures to a dataset.
Plans matter here. Monitors need capture, which starts on Starter. Starter includes 1 judge monitor, Team includes 5, and Enterprise is unlimited. Datasets and eval runs are on Team and Enterprise. The Free plan includes neither capture nor monitors.
We build TokenRouter. If you need deep tracing, experiments and CI gating, dedicated eval platforms do that well, and plenty of setups use both. Our monitor docs are at tokenrouter.io/docs/monitors.
Top comments (0)