Your agent waits a full second to send the number 3

Put a timer around the steps of any agent loop and the same shape shows up: a pause of one to three seconds, then a single small action. Click element 3. Keep the file. Stop waiting, the build is done.

In that pause a language model is writing something like:

Looking at the current page, I can see several navigation options. The most relevant to the user's goal appears to be the third link in the results list, which matches the destination they mentioned...

Your code parses that and keeps one number: 3.

A second of wall clock and a few hundred tokens, to transmit a value between 1 and 30. And the loop does it again on the next step, and the next.

If your first reaction is that this is just a badly configured call — constrain the output to an enum, cap the tokens, turn the thinking off — you are right, and it turns out to be worth most of the gap. I measured that too, and the number it leaves behind is further down.

Two kinds of steps

Go through what an agent loop actually asks for, step by step, and they sort into two piles.

Steps that must produce text. Write the commit message. Summarize the transcript. Generate the patch. Explain the failure. Only a language model does these.

Steps that only produce a decision. Is the build finished? Which of these 30 elements do I click? Is this shell command safe to run? Keep this message or drop it? Is this output a success?

LLMs do both piles, and they are structurally wasteful on the second: they answer a multiple-choice question by generating prose, which your code then throws away after extracting the answer.

What Jev is, before I sell you anything

Jev is TypeSafe's "System One" model, released four days ago. It takes a state plus typed questions — yes/no, pick-one, rate — and returns answers with confidences in a single forward pass. There is no token stream to wait for because it has nothing to generate.

The things you should know before adopting it, up front:

• It is API-only today — TypeSafe direct, OpenRouter, or the Vercel AI Gateway. There is no self-hosted path. The state you judge leaves your machine: page DOM, command output, transcript messages, the shell command you're asking about.
• It is weeks old, and so is everything around it, including what I built.
• It answers, it does not explain. If you need a reason in words, you still need an LLM.

I built jev-use on top of it: a plugin for Claude Code, Codex and pi that routes the no-text steps to Jev and leaves the LLM everything that actually produces writing.

The API

import { Jev, check, pick, rate } from "jev-use";

const jev = new Jev();

const { answers } = await jev.judge(ciRunSummary, {
  green: check("Did the run fully succeed?"),
  next:  pick("Next action?", { merge: "all green", rerun: "looks flaky", hold: "needs attention" }),
  risk:  rate("How risky?", ["routine", "worth a look", "incident"]),
});

answers.next.answer;      // "merge"
answers.next.confidence;  // 0.93
answers.next.escalate;    // false

Three questions about one state ride one call, and latency is flat in the number of questions, so there is no reason to ask them one at a time.

The other branch is the part that matters. Anything Jev can't or shouldn't decide comes back with escalate: true and a typed reason:

reason	meaning
writing	the step's output is text; the LLM has to do it
open_ended	the answer doesn't fit a yes/no, a choice, or a scale
oversized	the state is too big to judge honestly
unsure	it answered, but not decisively enough to act on
unreachable	the backend failed — the question goes back to the LLM

unreachable never resolves to a default answer. That distinction is load-bearing when the question was "is this command safe to run": a judge that can't be reached hands the question back, it does not wave the command through.

Three things I measured

Every number below comes from a script in bench/examples/ that you can re-run with your own key, and every GIF is a 1× recording of a real run with no cuts.

1. Decision rate — and what the number looks like after you fix the baseline

One paddle decision per ball step, three lanes, 20 seconds, the same three-option question to each model. The ball's speed on screen is the decision latency. Jev gets 86 decisions in 20 s; claude-haiku-4.5 gets 6 and gemini-3-flash 3 — all three called the way agent loops usually call them, which is the wrong way. Take the GIF as a feel for what decision latency looks like, not as a model comparison. The comparison is the table below.

That 14× is what a naive caller gets, and I don't think it's an honest model comparison — so I ran the fixed version before publishing it. Strict JSON-schema enum output on both baselines, and thinkingBudget: 0 for Gemini. 40 fresh states per arm, twice:

arm	p50	cost / 1k judgments
Jev	225 ms	$0.018
claude-haiku-4.5, called the way agent loops usually call it	2,874 ms	$1.67
claude-haiku-4.5, enum-constrained	691 ms	$0.30
gemini-3-flash, called the usual way	6,406 ms	$2.60
gemini-3-flash, enum-constrained, thinking off	1,027 ms	$0.09

The honest latency lead is 3×, not 14×. What survives the fair fight is cost — 5× cheaper than the constrained Gemini call and 16× cheaper than the constrained Haiku one — and that the answer is inside the option set by construction rather than parsed back out of prose. Note where that leaves you: a properly configured gemini-3-flash is the real competition, not the 6.4-second version of itself. On decision quality the five arms are indistinguishable: 24 to 30 correct out of 40 against a geometric reference, Jev included.

Two mechanism findings that cost me an afternoon, in case they save you one: on Haiku, reasoning_effort: "none" enables extended thinking (400+ reasoning tokens) where simply omitting the parameter gives zero; and Gemini's thinking only switches off through providerOptions.google.thinkingConfig.thinkingBudget: 0 — thinking_level and the top-level variants are silently ignored.

Two things you should see next to that 86. First, 69 of those decisions came back flagged unsure — through this gateway Jev returns no confidence field, so jev-use reconstructs it from the distribution margin, which is structurally thin on many-option questions; if your router escalates on unsure, the effective count is 17, not 86. Second, Jev answered stay zero times in 80 calls, including the 13 states where holding position was the right answer; its directional judgment on the other 27 was perfect. A model that never once picks one of your options fails silently, so check the answer distribution and not only the accuracy.

(All latencies here are client-side from a Linux container in Europe, network included. Yours will differ.)

2. A browser task where the two models split the work

OpenStreetMap directions, 20.7 seconds end to end. Jev makes all 10 click decisions (p50 274 ms); the LLM takes over at exactly the four moments where text goes into a field.

The interesting part isn't the click latency, it's a fault I didn't write into the script. OSM's geocoder resolves "Eiffel Tower" to a location 1,809 km from the intended one. It does that deterministically, which is why the repair reproduces in 8 runs out of 8 — a fixture, not luck, and the cleanest way I have to show the escalation path under a real failure. Jev's goal-level check ("does this route match what the user asked for?") rejected the route at 0.33 confidence and escalated. Because the fix is more text, the fields went back to the LLM with the geocoder's actual answer attached as evidence, and the second attempt verified at 0.94 on the real 3.7 km walking route.

3. Context compaction

A real transcript filling 94.6% of the context window. Jev judges 200 messages keep-or-drop in 7 batched calls (p50 408 ms); the LLM writes one 104-word paragraph to replace the dropped block. The window falls to 44.3% and 3 of 3 recall checks on the dropped facts still pass.

This one also produced the bug worth repeating to anyone building compaction: the first summary said "216 tests" where the source output said 58. The prompt now forbids the summarizer from computing numbers at all, and the demo traces every number in the paragraph back to a message on screen — 16 of 17 traceable in the run I kept, with the untraceable one flagged. If you compact context with an LLM, check its arithmetic.

One fair warning before you copy this pattern: compaction is the family Jev scored worst on in the accuracy run below — 56.3%. It works here because the keep-or-drop rule is written into the messages themselves. Read the next section before building on it.

Is it right, though?

Speed is the easy half. A 224 ms wrong answer is worth less than a 3 s right one, and until today every number in this repo was a latency number, with correctness samples of n=8.

So I measured it: 454 judgments over 422 real states across five families — shell commands an agent proposed to run, real command output with its real exit code, Hacker News rows, transcript messages judged keep-or-drop, and commits and PRs to triage. Nothing invented; every command was really run, every row really fetched.

Agreement with the reference	82.2% (373/454)
Escalated back to the LLM	14.1%
Agreement among the verdicts it acted on	89.5% (349/390)
Always-answer-the-majority-class baseline	68.7%
Whole corpus	$0.0051, 77 s
Same corpus, one claude-haiku-4.5 call per judgment	~$0.50, ~443 s (extrapolated from a random 45 of the 454)

89.5% is the operational number, because the escalated ones go back to the LLM by design — and escalation earns its place: of the 47 escalations that carried a provisional verdict, 51% would have been right, against 87.5% for the ones it acted on. (The other 17 escalations — writing, open_ended, oversized, unreachable — produce no answer to score.)

Now the parts that don't flatter it.

The grader is an LLM, and LLMs agree with LLMs. I scored against claude-opus-5 asked the identical typed question over the identical state. On a random 45-item subsample, claude-haiku-4.5 — weaker and far cheaper than the grader — agreed with it 35/45 where Jev agreed 33/45. Some of that gap is grader bias rather than Jev being worse, and this experiment cannot separate the two.

I hand-audited 34 reference labels and disagreed with 3. So roughly nine points of this is reference noise. Don't read any difference smaller than that, mine included.

It splits cleanly on whether the criterion is written into the state. Command completion, scored against real exit codes — actual ground truth, not an opinion — 73/73. Hacker News topical matching, 94.2%. Shell-command gating, 80.9% — and the direction of its errors is the part worth knowing. Of the 22 commands the reference labelled dangerous, Jev denied 18 and escalated 4: none were wrongly allowed. All four of its mistakes are over-refusals among the 88 safe ones, and all four are commands that mutate nothing (git merge nonexistent-ref, cp /nope ./x). It reads intent-to-mutate, not outcome. Note it clears that family's majority baseline by 0.9 points, i.e. not at all — read the per-class numbers, not the headline.

Context compaction was 56.3% — worse than always giving the same answer. That is the result I'd want to know before adopting this, so: don't hand Jev a keep-or-drop rule that lives in your head rather than in the state. (The honest counter-caveat is that 29 of that family's 38 disagreements trace to one reference decision at a batch boundary, so its effective sample is closer to 10 than 87 — which cuts both ways.)

Corpus, provenance, per-family confusion patterns, verbatim disagreements and the run-to-run variance are in bench/RESULTS.md; node bench/examples/agree.mjs re-runs the whole thing.

Where this does not help

• One-off judgments. Saving 300 ms once is invisible. This only pays inside loops that make the same kind of decision over and over.
• Loops that are mostly generation. If nearly every step writes code, the router hands nearly everything back to the LLM and you have saved nothing. yoshi measured −0.03% on exactly that shape.
• Retroactive context pruning. Judging old context for "will I need this later" dropped later-needed information 73% of the time in pi-jev-context's measurements, and stays shadow-only in that project. Judge at write time or not at all.
• Decisions a 20-line heuristic already makes. If a regex or a local 0.5B classifier answers your question at 20 ms for nothing and never leaves the machine, use that. This is for the open-ended states you cannot enumerate — and for those, a round trip to a third party is the price.

All of the caveats, the variance, and the runs that went badly are in bench/RESULTS.md.

Try it

npx -y jev-use install     # wires Claude Code, Codex and pi — whichever it finds

That command does not hand-edit your config. It runs each harness's own registration command — claude mcp add --scope user jev -- npx -y jev-use serve, the Codex equivalent, pi install — so you undo it the same way you undo any other MCP server: claude mcp remove jev.

Then set TYPESAFE_API_KEY, OPENROUTER_API_KEY or AI_GATEWAY_API_KEY for whichever provider you want, or JEV_BACKEND=mock for a keyless dry run that makes no network calls at all. npx -y jev-use doctor prints what it found.

One thing the install line doesn't say, and it matters. As an MCP server, the harness still spends an LLM turn deciding to call jev_judge — so what the plugin buys you there is a cheaper, guaranteed-in-set answer and the ability to batch thirty questions into one call, not a faster individual step. The two paths that actually delete LLM latency are the PreToolUse gate, which runs as a hook with no model turn at all, and the library: npm i jev-use, your own loop calling jev.judge(), no runtime dependencies on the judgment path. The 3× in this post is the library path. Use the plugin for cost, gating and batching; use the library if you want the latency.

Repo, scripts, and the full measurements: github.com/shitianfang/jev-use (MIT).

The code was substantially written with Claude Code. The measurements are mine, and every one of them re-runs from the repo.