Every answer-verification vendor publishes a benchmark. Every one of them wins it.
So we ran thirteen systems on a single test set — 2,018 items, identical labels, published scoring
code — and put the results in one table. Two things fell out that are worth your time even if you
never use any of these products.
One: a reference row that looks at nothing but answer length and formatting scores 0.7036,
which is above eight of the thirteen systems measured.
Two: the system that wins the ranking is not the system that helps an agent most. Those turned
out to be different questions with different answers.
What a typed decision model is
A typed decision model returns a structured verdict — a boolean, a choice, an ordinal score —
instead of free text. One forward pass, zero generated tokens. That is what makes it one to
three orders of magnitude cheaper than asking a frontier model the same question.
TypeSafe AI's Jev established the category under the name System One models. An ecosystem has
grown around it: open reproductions, architecture-compatible alternatives like Convai's Laya,
adjacent tools like Patronus Lynx and Vectara HHEM.
What did not exist was a single test set all of them had been run on.
The setup
| Task | Given a question and an answer written by some model, score whether the answer is correct |
| Items | 2,018 · 508 incorrect · 5 domains · answers from 4 different models |
| Metric | AUC — how well wrong answers sort to the bottom. Threshold-free. 0.5 = coin flip |
| Aggregation | Per domain first, then size-weighted |
| Uncertainty | Paired bootstrap, 3,000 resamples. An interval containing zero yields no rank |
Two choices change what the numbers mean, so they belong up front rather than in a methods
appendix.
Per-domain aggregation. Pooling all 2,018 items into one AUC gives systematically higher
numbers, because score scales differ between domains and pooling rewards that difference rather
than discrimination. On our own system the pooled figure reads 0.80 where the per-domain mean
reads 0.73. We report the lower one.
Published baselines. Two reference rows sit in the table: a logistic model over ten surface
features of the answer (length, digit count, formatting), and the answering model's own stated
confidence. They are there so every other number has something to be measured against.
Results
| # | System | Vendor | Local weights | AUC |
|---|---|---|---|---|
| 1 | ZTC (397B) | VIDRAFT | yes | 0.7364 |
| 2 | Jev | TypeSafe AI | no | 0.7350 |
| 3 | ZTC (27B) | VIDRAFT | yes | 0.7282 |
| — | Length & formatting baseline | reference | — | 0.7036 |
| 4 | open-jev 4B | pngwn | yes | 0.6844 |
| 5 | Patronus Lynx 8B | Patronus AI | yes | 0.5179 |
| 6 | Laya-Typed-Decisions 421M | Convai | yes | 0.5144 |
| — | Model's own stated confidence | reference | — | 0.5000 |
| 7 | Laya-Multilingual 322M | Convai | yes | 0.4796 |
Asking an LLM directly (generates tokens, so outside the axis, unranked): GPT-5.2 0.7148 ·
Qwen3-Next-80B 0.6366 · GPT-4o-mini 0.5878 · Gemini 2.5 Flash-Lite 0.5822.
Different axis: Vectara HHEM-2.1 0.4852 — it decides whether an answer follows from a supplied
document, and this test set supplies none.
The top of the table is a tie
Our 397B leads Jev by 0.0014. The 95% interval on that gap runs from −0.019 to +0.032.
Under the board's own rule — an interval containing zero produces no rank — first and second are
not distinguishable. That is stated above the table, not in a footnote. A leaderboard whose
error bars only appear when someone else wins is not a leaderboard.
Four findings that survive the error bars
1. The surface baseline is the real bar
Length and formatting alone reach 0.7036. Eight of thirteen systems score below it.
A verifier under that line is not detecting correctness. It is detecting shape — longer answers
with more structure tend to be right, and you can learn that without understanding anything.
Any leaderboard in this category that omits the baseline is flattering its entrants. If you are
evaluating a verifier for your own stack, fit this baseline on your own data first. It costs about
twenty lines and it will disqualify most candidates.
2. Bigger is not better on this axis
Within the same family, the 397B beats the 27B overall. On scientific reasoning the order reverses
hard:
scientific reasoning 27B 0.7410 · 397B 0.6287
A model fourteen times larger scores 0.11 lower. This lines up with a separate observation —
a 180B-class model scoring 0.7146, below a 4B at 0.7284.
Verification quality tracks representation geometry, not parameter count. Scaling the judge is not
a strategy.
3. "Just ask a frontier model" is expensive and mid-table
GPT-5.2 reaches 0.7148 — below both typed-decision configurations and below Jev — while generating
tokens and costing roughly $0.55 per 1,000 calls against Jev's $0.024. Twenty-three times
the price for a lower score.
Smaller judges fall further: GPT-4o-mini 0.5878, Gemini 2.5 Flash-Lite 0.5822 — both under the
length baseline.
4. An open alternative exists, and its speed claim holds
Laya is architecturally the closest open analogue: non-autoregressive, zero generated tokens,
Apache-2.0, ~0.015 s per call — roughly 140× faster than the hosted API we measured. The speed
is real.
The accuracy on this set is not: 0.4796 / 0.5144, against a published claim of beating Jev on the
vendor's own English typed-decision benchmark. Both numbers can be true. They are different
tests — which is precisely the argument for a shared one.
The part that actually matters: AUC is not the number you deploy on
The ranking answers "which system separates right answers from wrong ones best." It does not
answer "which system helps my agent."
We measured the second question directly. For each item: score the answer with a verifier, send the
lowest-scoring k% to a stronger model to be re-answered, keep everything else. All arms draw from
one shared pool of re-answers, so no arm gets luckier retries. Ties broken randomly, averaged over
200 seeds.
At a 20% retry budget, against a 74.83% no-gate baseline:
| Gate | Final accuracy | vs. no gate |
|---|---|---|
| ZTC | 76.16% | +1.34 pp |
| Jev | 74.76% | −0.07 pp |
| Random | 74.58% | −0.25 pp |
The two systems differ by 0.0014 AUC and by 1.4 points of end-to-end accuracy.
Why — and this is the transferable part
Re-answering is double-edged:
wrong answers sent back → 38% get fixed
right answers sent back → 30% get broken
A gate's value is therefore set by precision, not recall. Sending back a correct answer is not
free; it is a coin flip weighted slightly in your favour.
At a 20% budget, Jev routed 403 items of which 216 were already correct. ZTC routed 403 of
which 195 were already correct. Twenty-one items of difference in what got sent back, and one
arm repaired about as much as it damaged.
That asymmetry is completely invisible in an AUC column.
Scope: one escalation target, one item set. We report it as a mechanism, not a universal
constant. The direction of the effect should transfer; the magnitude is ours, not yours.
What we publish and what we cannot
Published in full: every score, every label, the grading code.
Not redistributed: the source items. They come from corpora whose licences forbid
redistribution or are access-gated, and from commercial model outputs whose terms do not clearly
permit republication. The grading code is published so the identical protocol runs against any
private set.
Listed without a score: four published reproductions that would not execute from their released
artefacts — a missing classifier head, unsupported architectures, an incomplete tokenizer. Each
appears on the board with the failure and a link, because a category's real state includes what
does not run.
Marked pending: one entry whose re-measurement disagreed with its earlier value by a margin our
harness cannot yet account for. It keeps its prior number, flagged, until the harness matches the
upstream implementation. Lowering a competitor's score on the strength of our own untrusted code is
not a correction.
If you take three things
- Fit the surface baseline before you evaluate anything. Length and formatting reach 0.7036 here. Most verifiers do not clear it.
- Rank on the metric you deploy on. A 0.0014 AUC gap produced a 1.4-point swing in agent accuracy, in the opposite direction from what the ranking implied.
- Precision, not recall, is what a gate is for — because sending a correct answer back to be redone breaks it 30% of the time.
Leaderboard: https://huggingface.co/spaces/mayafree/typed-decision-leaderboard
To add a system, open a discussion on the Space with a link and a runnable scoring snippet. Entries
that cannot be executed from published artefacts are listed under did not run rather than omitted.
Top comments (0)