DEV Community

Ahab
Ahab

Posted on Originally published at indieseek.co

Jev benchmark: reproduce latency, cost, calibration, and decision quality before rollout

Jev benchmark: reproduce latency, cost, calibration, and decision quality before rollout

Quick answer

Do not approve Jev from a vendor chart or a single fast request. Freeze your own cases, workflow code, model ID, pricing snapshot, region, and comparison settings. Record every request's resolved model, token usage, wall-clock latency, probabilities, confidence, final action, review outcome, and error. Then judge five things together: action quality, calibration, tail latency, accepted-decision cost, and review load.

TypeSafe currently offers Jev in early access. Its official workflow eval reports strong accuracy, time, and cost results across four example workflows, but TypeSafe also documents important boundaries: the workflow authors work on its model-capabilities team, the harness is assumed correct, reference answers come from GPT-6 Astra and Claude Fable 5.1 at high thinking, and published latency is measured from the service's current region. Those are useful hypotheses—not your production evidence.

Who this is for

This guide is for teams that have already chosen a narrow typed-decision workload and built a safe fallback. If you are still choosing a model, use the Jev workload decision guide. If your Choice, Score, Noul, and review states are not explicit yet, start with the typed fallback workflow.

The goal here is not to reproduce TypeSafe's entire benchmark. It is to answer a smaller question: does Jev improve your accepted outcomes enough to justify a controlled rollout?

Freeze a benchmark manifest

Version the corpus and policy before sending the first request:

corpus: support-triage-v4
cases_sha256: 4f2e…
policy_commit: 91ac…
question_set: triage-2026-09-27
model_requested: jev-1.13.0
comparison_model: your-current-production-model
region: ap-southeast-client
price_snapshot: 2026-09-27
repetitions: 3
Enter fullscreen mode Exit fullscreen mode

Use de-identified historical cases with the real class balance and named slices: easy, ambiguous, rare, high-impact, long-state, non-English, and API-failure fixtures. Split threshold tuning from the final holdout. Do not rewrite criteria after seeing holdout errors; that turns evaluation into prompt fitting.

Run candidates against the same state, questions, deterministic policy, concurrency, timeout, and retry budget. Randomize run order to reduce time-of-day bias. Pin jev-1.13.0 rather than jev-latest; TypeSafe warns that aliases move.

Store one receipt per attempt

Write an append-only JSONL row even when the call fails:

{"case":"c-018","attempt":2,"requested_model":"jev-1.13.0","resolved_model":"jev-1.13.0","input_tokens":1840,"latency_ms":412,"http_status":200,"decision":"review","reviewed_label":"review","probabilities":{"auto":0.18,"review":0.77,"deny":0.05},"confidence":0.69,"policy":"risk-v3"}
Enter fullscreen mode Exit fullscreen mode

Use a monotonic clock from client send to validated response. Report p50, p95, and p99, plus timeout, 429, 5xx, schema-validation, and retry rates. Keep cold starts, warm traffic, payload-size bands, concurrency, and client region separate. A provider's server-side number cannot replace user-observed tail latency.

Measure decisions and uncertainty

Score the final workflow action, not just the model primitive. Use exact match where every action has equal cost, or a predeclared loss matrix when a false auto-approval is worse than an unnecessary review. Publish per-slice results and bootstrap confidence intervals; a blended average can hide a dangerous minority case.

For Noul, calculate Brier score: mean((p - y)^2). For Choice, use the multiclass Brier score over every option. For Score, keep the full level distribution and report an ordinal proper score or level error alongside action loss. For Choice and Score, also plot observed accuracy by confidence bin and report expected calibration error. Calibration is useful only if higher reported certainty actually predicts higher correctness on your data.

Calculate accepted-decision cost

TypeSafe's current first-party price is $0.042 per million input tokens, with output currently free. Snapshot that price rather than hard-code it forever.

inference_cost = sum(input_tokens × price_per_input_token)
review_cost = review_minutes × loaded_reviewer_rate / 60
accepted_decision_cost = (inference_cost + review_cost) / correct_automatic_decisions
Enter fullscreen mode Exit fullscreen mode

Also report cost per case, abstention rate, review rate, retry cost, and the denominator. A cheap model that sends most cases to people—or confidently automates the wrong cases—can have a worse accepted-decision cost.

Use a release gate, not a leaderboard

Set the gate before unblinding the holdout. Example placeholders:

Gate Pass condition
High-impact loss No regression beyond the predeclared tolerance
Calibration Brier/ECE improves or stays within tolerance in every critical slice
Tail latency p95 and timeout rate remain within the product budget
Accepted-decision cost Beats the production baseline after review labor
Review load Fits staffed capacity without hiding errors
Reliability 429/5xx/retry rates stay below the operational ceiling

Start in shadow mode, then canary a small reversible slice. Roll back on a loss-bound breach, calibration drift, error-rate spike, review-queue overload, or unexpected model resolution. Promotion requires a new signed manifest; changing the model, question set, policy, thresholds, or price snapshot invalidates the old gate.

Common mistakes

  • Treating TypeSafe's published 193.6× speed or 444.6× cost headline as a universal result.
  • Comparing Jev at one setting with a competitor at a different reasoning, retry, or workflow setting.
  • Tuning thresholds on the holdout or removing hard cases after failures.
  • Reporting mean latency without tails and errors.
  • Calling confidence “calibrated” without reliability bins and proper scores.
  • Dividing spend by all requests instead of correct automatic decisions.
  • Shipping because schema validity passed while action quality or review capacity failed.

Make your Mac notch useful with SuperNotch—22 native tools for music, clipboard, focus, screenshots, system controls, and more.

FAQ

Did IndieSeek independently reproduce TypeSafe's numbers here?

No. This article verifies the current official methodology, API, model, and price documentation, then provides a reproducible protocol. It does not claim an authenticated independent benchmark run.

How many cases are enough?

There is no universal number. Size the holdout around the rare, costly errors you must detect, then report uncertainty intervals. A large easy set cannot compensate for too few high-impact examples.

Should the vendor eval be ignored?

No. Use it to form hypotheses and understand the proposed workflow shape. Production approval should depend on a frozen, workload-specific comparison that your team can replay.

Sources

Originally published on IndieSeek.

Top comments (0)