A support agent I work on gets graded every night against ten thousand past tickets. One morning the run came back red. Nobody had merged a prompt change, the model version was pinned, and the retrieval index had not been rebuilt. I spent most of that morning diffing configs. The answer was in the judge logs: the same reply to the same ticket had scored 0.7 the night before and 0.6 that night, and my pass mark was 0.7. The agent was identical on both nights. The judge had scored the same input two different ways.
I keep relearning this one. An LLM judge gives you a slightly different score each time you run it, so on a gate that grades thousands of cases, some of the red is the judge disagreeing with itself.
A regression gate needs the same verdict twice
I like LLM-as-a-judge for what it does well. You hand a strong model a rubric, it returns a score and a sentence explaining the score, and that sentence is the most useful thing I have when I am debugging one bad answer. I can read it, argue with it, and log it.
A gate asks for something else. It has to return the same verdict on the same input every time, so that a failing case means the agent regressed. Code checks and semantic verifiers return the same result on the same input every time, so they can hold a gate on their own. For the LLM judge I needed a workaround, and mine was to average several judge runs on the cases that mattered. That multiplied the cost of the slowest and most expensive check in the suite.
What a typed judge gives up to get there
JEV is TypeSafe AI's typed evaluator, released in September 2026. Its makers frame it as a "System One" model built for fast, instinctive scoring. The difference you notice first is the output. JEV returns a typed verdict (a yes or no, a multiple-choice pick, or a point on an ordered scale) with a calibrated probability attached. It writes no rationale at all.
The numbers come from the paper "JEV-as-a-Judge: Accept When Confident, Escalate When Unsure". In its September 29 revision, JEV recorded a 0.15-second median latency and cost 0.36% of the strongest comparator's fee on the tested workloads. It came within three percentage points of GPT-6 when the verdict could be read directly from the supplied text. The gap widened on math, code, logic and other tasks where the verdict has to be derived. I treat those figures as the paper's benchmark setup. My own traffic will land somewhere else, and I would only trust a number I measured on it.
So JEV gives up the written reasoning to get repeatability. For a regression suite that runs every night, I will take that trade.
The probability is the part I use most
The calibrated probability changes how I wire the judge in. I accept the verdict when confidence is high and route the low-confidence cases to a stronger reasoning judge or to a person. That makes the typed judge a triage step in front of the expensive checks.
The threshold is mine to set. I validate it against my own labeled cases before it is allowed to block anything, because a cutoff copied from someone else's benchmark describes their data.
One agent, four checks, four methods
I pick each method by what the check is grading. On the support agent, that splits four ways.
Whether the reply is valid JSON with the required fields is a format question. A code check owns it: deterministic, instant, free.
Whether the reply matches the known resolution for that ticket is agreement with a reference. A semantic verifier handles it cheaply. You embed the reply and the reference answer, take the cosine similarity, and pass it above a threshold. A small model like all-MiniLM-L6-v2 runs locally. I started at 0.75 and tuned from there against known-good and known-bad pairs, since too low lets paraphrases of a wrong answer through and too high fails correct answers worded differently.
Whether the tone stayed polite and on-brand is a judgment call, and that is where an LLM judge earns its cost. I keep it pointwise, one answer scored against a threshold. When I need a ranking I switch to pairwise and run both orders, because the model tends to favor whichever answer it reads first.
The nightly run over ten thousand past tickets is the high-volume gate, and that is where the typed verdict fits.
Where I still want a paragraph of reasoning
Two places keep a reasoning judge or a human in my loop.
The first is derivation. If the verdict depends on checking math, code, or a multi-step argument, the typed judge falls further behind, so those cases escalate by default.
The second is adversarial input. Typed and generative judges both read the content they grade, and that content can carry instructions aimed at the judge. A written rationale does not reliably expose prompt injection, and a typed verdict gives me even less to investigate after the fact. When a judge feeds a security or moderation decision, I treat its input as untrusted, keep the evaluation instructions isolated, test adversarial examples, use a deterministic rule wherever the policy can be written down exactly, and send uncertain cases to human review.
What I would tell myself before that red morning
- Gating on a judge with variance. A red build that does not reproduce may be judge variance. Put a low-variance check on the gate.
- Reading similarity as truth. "Paris is the capital of France" and "Paris is not the capital of France" score as very similar. Anywhere one flipped word changes the verdict, pair the verifier with a rule or a judge.
- Borrowing a confidence threshold. Calibrate it on your own labels first.
- Asking one judge to grade everything. Formats, references, tone and volume each have a method that fits. Most production agents need three or four of them running together.
The full comparison, with the decision table and Python examples for the semantic verifier and the LLM-judge pattern, is in this piece.
Which check in your suite has gone red with nothing changed underneath it? For me it was the nightly regression gate, and the fix was handing that job to a check that gives close to the same answer every time.
Top comments (0)