AI OpenFree

Posted on Sep 20

JEV vs ZTC: Statistically Tied on Accuracy, 1.4 Points Apart Where It Counts

Short answer: on accuracy, JEV (0.7350 AUC) and ZTC (0.7364) are statistically tied — the 0.0014
gap has a 95% interval of −0.019 to +0.032. They separate on two other axes: deployment shape (API
only vs open weights) and downstream agent accuracy, where ZTC gained 1.34 points and JEV lost 0.07
at the same retry budget.

Both numbers come from the same test set: 2,018 items, 508 of them incorrect, identical labels,
published grading code —
Typed Decision Leaderboard.

First, what these two things are

JEV is TypeSafe AI's answer verifier, the product that established this category under the name
System One models. ZTC (Zero-Token Confidence) is VIDRAFT's, shipping as two open-weight
models.

Both are typed decision models: they return a structured verdict — boolean, choice, or score —
from a single forward pass, emitting zero generated tokens. That is what makes both of them one
to three orders of magnitude cheaper than asking a frontier model "is this answer right?"

So the choice between them is not a choice of architecture. It is a choice of accuracy, shape, and
behaviour under load.

Axis 1 — accuracy: a tie, and it should be reported as one

System	AUC	95% interval vs. the other
ZTC (397B)	0.7364	—
JEV	0.7350	−0.019 to +0.032
ZTC-Judge-27B	0.7282	−0.034 to +0.020

The interval contains zero in both comparisons. Under the rule that a gap indistinguishable from
noise produces no rank, neither system can be called more accurate than the other on this set.

For context, the same table's reference row — a logistic model over answer length and formatting —
scores 0.7036, and eight of the thirteen measured systems fall below it. Both JEV and ZTC are
comfortably above that bar. Most of the category is not.

Axis 2 — deployment shape, where they genuinely differ

This is the axis that usually decides the purchase, and it is not close.

	JEV	ZTC
Weights published	no	yes (Apache-2.0)
Runs inside a private network	no	yes
Adaptable to your own data	no	yes
Hosted API	yes	yes
Cost	$0.024 / 1,000 calls	your own compute
Generated tokens	0	0

If your data cannot leave your network, the accuracy tie is irrelevant — only one of these is
installable. If you would rather not operate a GPU, the same logic runs the other way, and $0.024
per thousand calls is cheap enough that "just use the API" is a defensible answer.

For comparison, asking GPT-5.2 the same question costs roughly $0.55 per 1,000 calls — about
23× more — and scores 0.7148, below both.

Axis 3 — per domain, the ranking inverts

The headline is an average of five domains. They disagree.

Domain	n	JEV	ZTC 397B	ZTC 27B
Professional exams (law/math/bio)	400	0.7981	0.8660	0.8462
Biology & medicine	917	0.7185	0.7433	0.7154
Commonsense & multi-step	278	0.6006	0.6072	0.6172
Disaster & safety guidance	225	0.7620	0.7319	0.6961
Scientific reasoning	198	0.8421	0.6287	0.7410

JEV wins scientific reasoning outright at 0.8421 — 0.21 ahead of the overall leader. If that is
your workload, the overall ranking is the wrong table to read, and the answer to "which one" is JEV.

Two other things in that table are worth carrying away regardless of vendor:

Bigger is not better. Inside one family, the 397B scores 0.11 below the 27B on scientific
reasoning. A separate measurement puts a 180B-class model at 0.7146, under a 4B at 0.7284.
Verification quality tracks representation geometry, not parameter count.

Commonsense is unsolved by everyone. Every system sits near 0.60 on multi-step commonsense. No
product on this board fixes that domain, and knowing it is more useful than any ranking.

Axis 4 — as an agent gate, they are not tied at all

AUC answers "which separates right from wrong answers best." It does not answer "which helps my
agent." Those turned out to be different questions.

Setup. For each item: score the answer with a verifier, send the lowest-scoring 20% to a
stronger model to be re-answered, keep the rest. All arms draw from one shared pool of re-answers,
so no arm gets luckier retries. Ties broken randomly, averaged over 200 seeds.

Gate	Final accuracy	vs. no gate (74.83%)
ZTC	76.16%	+1.34 pp
JEV	74.76%	−0.07 pp
Random	74.58%	−0.25 pp

A 0.0014 AUC gap became a 1.4-point swing in end-to-end accuracy. At this budget, JEV's gate
performed about the same as no gate at all.

The mechanism, which is the transferable part

Re-answering is double-edged:

wrong answers sent back  →  38% get fixed
right answers sent back  →  30% get broken

So a gate is not rewarded for catching errors. It is rewarded for not disturbing what was already
right. Precision, not recall — the opposite of how most people reason about a safety net.

At a 20% budget both systems routed 403 items. JEV's 403 contained 216 that were already
correct; ZTC's contained 195. Twenty-one items of difference in what got sent back, and one
arm spent its whole retry budget repairing about as much as it damaged.

None of that is visible in an AUC column, and AUC is what every vendor publishes.

Scope, stated plainly: one escalation target, one item set. The direction should transfer; the
magnitude is theirs, not yours. Run the same protocol on your own data — the grading code is
published for exactly that.

How to choose

If this is true for you	Pick
Data cannot leave your network	ZTC (only one with published weights)
You want to fine-tune the verifier on your own data	ZTC
Your workload is scientific reasoning	JEV (0.8421, clear win)
You do not want to operate a GPU	JEV ($0.024 / 1,000)
You are gating agent retries	Measure both on your own set — this is where they diverge
You are choosing on the leaderboard number alone	Don't. The gap is noise.

Frequently asked questions

Is ZTC better than JEV?

Not on accuracy. The two are statistically tied on the shared test set — 0.7364 versus 0.7350, with
a 95% interval spanning zero. They differ on deployment shape (ZTC publishes weights, JEV does not),
on per-domain strengths (JEV wins scientific reasoning at 0.8421), and on downstream agent accuracy
at a 20% retry budget (+1.34 pp versus −0.07 pp).

Is there an open-source alternative to JEV?

Yes. ZTC ships two Apache-2.0 models with published weights —
ZTC-Judge-27B (verification only) and
Darwin-397B-ZTC (a generation model with the
confidence probe on board). Both score above the 0.7036 surface baseline. Other open entries score
lower: open-jev 4B at 0.6844, Laya at 0.4796 / 0.5144.

How much does JEV cost compared with running a verifier yourself?

JEV is about $0.024 per 1,000 calls. A self-hosted verifier costs only compute you already own.
Asking GPT-5.2 the same question costs roughly $0.55 per 1,000 calls and scores lower than
either.

What AUC should I expect from a good answer verifier?

Use 0.7036 as the bar, not 0.5. That is the score from answer length and formatting alone, and
eight of thirteen measured systems fall below it. JEV and ZTC are the only two systems above 0.73.

Can I test JEV and ZTC on my own question?

Yes — the Verifier Playground runs
ZTC, JEV and Laya on the same input, in the browser, no signup. One case cannot rank them, but it
shows you how each behaves on your phrasing.

Why did the more accurate verifier perform worse as a gate?

Because gating rewards precision rather than recall. Sending a correct answer back to be re-answered
breaks it about 30% of the time, so routing a few more already-correct items cancels out the errors
you caught. JEV routed 216 already-correct items; ZTC routed 195.

DEV Community