DEV Community

AI OpenFree
AI OpenFree

Posted on Originally published at huggingface.co

Which LLM Answer Verifier Is Most Accurate? 13 Systems, One Test Set, Published Labels.

Short answer: ZTC (397B) at 0.7364 and JEV at 0.7350 are the two most accurate answer verifiers
measured, and the gap between them is not statistically distinguishable. A baseline that reads only
answer length and formatting scores 0.7036 — above eight of the thirteen systems tested.

Full table and per-domain scores:
Typed Decision Leaderboard.

Every vendor in this category publishes a benchmark and wins it. This board exists because none of
those benchmarks shared a test set.

What is being measured

Task Given a question and an answer written by some model, score whether the answer is correct
Items 2,018 · 508 of them incorrect · 5 domains · answers from 4 different models
Metric AUC — how well wrong answers sort to the bottom. Threshold-free. 0.5 = coin flip
Axis Factual verification, no grounding document supplied
Aggregation Per domain first, then size-weighted
Uncertainty Paired bootstrap, 3,000 resamples. Interval containing zero → no rank assigned

Two methodology choices change what the numbers mean, so they belong here rather than in an
appendix.

Per-domain aggregation. Pooling all 2,018 items into a single AUC produces systematically higher
figures, because score scales differ between domains and pooling rewards that difference instead of
discrimination. On the board's own leading system the pooled figure reads 0.80 where the
per-domain mean reads 0.73. The lower, reproducible one is what gets published.

Baselines are in the table, not implied. A logistic model over ten surface features of the answer
— length, digit count, formatting — and the answering model's own stated confidence both appear as
rows, so every other number has something to be measured against.

The ranking

| # | System | Vendor | API | Local weights | AUC |
|---|---|:---:|:---:|---|
| 1 | ZTC (397B) | VIDRAFT | yes | yes | 0.7364 |
| 2 | JEV | TypeSafe AI | yes | no | 0.7350 |
| 3 | ZTC (27B) | VIDRAFT | yes | yes | 0.7282 |
| — | Length & formatting baseline | reference | — | — | 0.7036 |
| 4 | open-jev 4B | pngwn | no | yes | 0.6844 |
| 5 | Patronus Lynx 8B | Patronus AI | yes | yes | 0.5179 |
| 6 | Laya-Typed-Decisions 421M | Convai Innovations | no | yes | 0.5144 |
| — | Model's own stated confidence | reference | — | — | 0.5000 |
| 7 | Laya-Multilingual 322M | Convai Innovations | no | yes | 0.4796 |

Just asking an LLM (generates tokens, so outside the axis and unranked): GPT-5.2 0.7148 ·
Qwen3-Next-80B 0.6366 · GPT-4o-mini 0.5878 · Gemini 2.5 Flash-Lite 0.5822.

Different axis: Vectara HHEM-2.1 0.4852 — it judges whether an answer follows from a supplied
document, and this test set supplies none. Reported outside the ranking with the reason.

First and second place are tied, and the board says so

The leading system is ahead by 0.0014 AUC. The 95% interval on that gap runs from −0.019 to
+0.032, so under the board's own rule no rank is assigned between them. The 27B-versus-JEV
interval is −0.034 to +0.020 — also containing zero.

That statement sits above the table rather than in a footnote. Error bars that appear only when
someone else is winning are not error bars.

Per-domain, where the ranking falls apart

The headline is an average of five domains. The domains disagree:

Domain n JEV ZTC 397B ZTC 27B Lynx 8B
Professional exams (law/math/bio) 400 0.7981 0.8660 0.8462 0.5454
Biology & medicine 917 0.7185 0.7433 0.7154 0.4816
Commonsense & multi-step 278 0.6006 0.6072 0.6172 0.5207
Disaster & safety guidance 225 0.7620 0.7319 0.6961 0.5459
Scientific reasoning 198 0.8421 0.6287 0.7410 0.5945

Two things fall out of that last row.

JEV wins scientific reasoning outright, by 0.21 over the top-ranked system. If your workload is
scientific reasoning, the overall ranking is the wrong table to read.

Bigger is not better. Within one family the 397B scores 0.11 below the 27B on that domain — a
model fourteen times larger, eleven points worse. This matches a separate measurement putting a
180B-class model at 0.7146, below a 4B at 0.7284. Verification quality tracks representation
geometry, not parameter count.

And note the commonsense row: every system sits near 0.60. That domain is unsolved by all of
them, which is more useful to know than any ranking.

What the board publishes that most leaderboards do not

Systems that would not run. Four published reproductions are listed with no score and the actual
failure:

Entry Why it did not run
heman10x/rlcd-modernbert-151m GLiClass architecture — transformers does not recognise it
heman10x/openJev-verdict-2.0 same
com-kotobalabs/open-jev-deberta-v3-large classifier head absent from the checkpoint → randomly initialised
AlexWortega/openjev released artefacts incomplete

They appear on the board because a category's real state includes what does not execute from its
published artefacts. Silently omitting them would make the field look healthier than it is.

One entry marked pending. Its re-measurement disagreed with the earlier value by a margin the
reproduction harness cannot yet account for, so it keeps its prior number, flagged, until the
harness matches the upstream implementation. Lowering a competitor's score on the strength of your
own untrusted code is not a correction.

Data availability, stated precisely. Scores, labels and grading code are published in full. The
source items are not redistributed — they come from corpora that are CC BY-ND, unlicensed, or
access-gated, plus commercial model outputs whose terms do not clearly permit republication. The
grading code is published so the identical protocol runs against any private set.

Frequently asked questions

Which answer verifier is the most accurate?

ZTC (397B) at 0.7364 and JEV at 0.7350, and the difference between them is not statistically
distinguishable (95% interval −0.019 to +0.032). ZTC (27B) follows at 0.7282.

Is Jev accurate at verifying LLM answers?

Yes. JEV scores 0.7350, well above the 0.7036 surface-feature baseline, and it is the strongest
closed system measured. It also wins scientific reasoning outright at 0.8421. Its limitation is
deployment shape rather than accuracy — the weights are not published, so it cannot run inside a
private network or be adapted to your own data.

Is there an open-source alternative to Jev?

Yes, with a real accuracy gap. open-jev 4B reaches 0.6844. Laya (Apache-2.0, 322M/421M) is
architecturally the closest open analogue — non-autoregressive, zero generated tokens, ~0.015 s per
call — but scores 0.4796 / 0.5144 here. Its publisher reports beating Jev on their own English
benchmark; that result did not reproduce on this test set. Both can be true: they are different
tests, which is the argument for a shared one.

How much does answer verification cost?

JEV costs about $0.024 per 1,000 calls. Self-hosted verifiers cost only compute you already own.
Asking GPT-5.2 the same question costs roughly $0.55 per 1,000 calls — about 23× more — and
scores lower than both leaders.

Can I just ask a large LLM instead of using a verifier?

You can, and it is measured. GPT-5.2 reaches 0.7148, below both leading verifiers, while generating
tokens and costing far more per call. Smaller judges fall further: GPT-4o-mini 0.5878, Gemini 2.5
Flash-Lite 0.5822 — both below the length-and-formatting baseline.

What is a good AUC for an answer verifier?

Use 0.7036, not 0.5. That is the score from answer length and formatting alone. A verifier below
that line is not detecting correctness; it is detecting surface shape. Fitting this baseline on your
own data takes about twenty lines and will disqualify most candidates before you read a datasheet.

Why is the top entry not declared the winner?

Because the interval around the first-to-second gap contains zero, and the rule that a gap
indistinguishable from noise produces no rank is applied at the top of the table as well as the
bottom.

Adding a system

Open a discussion on the Space with a link to the model or API and a runnable scoring snippet.
Systems that cannot be executed from their published artefacts are listed under did not run
rather than omitted.

Top comments (0)