Most numbers you'll see about TypeSafe's Jev decision model come from TypeSafe. I wanted my own, so I ran Jev 1.13, Mistral Small 3.2, Gemini 2.5 Flash Lite and GPT-5 nano through the same eight fixtures on one gateway, five runs each (40 per model), on 2026-09-20.
Headline: cheaper and faster, but not by the multiples you've heard.
| Model | Median | Mean cost | vs Jev |
|---|---|---|---|
| Jev 1.13 | 352 ms | $0.0000188 | 1x |
| Mistral Small 3.2 | 1,343 ms | $0.0000255 | 1.4x |
| Gemini 2.5 Flash Lite | 877 ms | $0.0000323 | 1.7x |
| GPT-5 nano | 7,504 ms | $0.0003434 | 18.3x |
Against a cheap chat model, Jev is 1.4 to 1.7x cheaper, not 100x. The big multiples only show up against reasoning models (GPT-5 nano, almost all of it thinking tokens) and frontier models: on one fixture, GPT-5.6 Sol cost 32.5x, GPT-5.6 Terra 34.3x and Opus 5 184.9x what Jev did.
Accuracy: Jev doesn't win. On 27 support tickets with a clear correct label, three passes each, Jev 1.13 and Mistral Small 3.2 both scored 27/27; Gemini 2.5 Flash Lite and GPT-5 nano scored 26/27. "Jev is more accurate" isn't supported by this.
What Jev does have that the chat models don't is a confidence number. Its confidence was 0.979 on the clear tickets and dropped to 0.841 on six deliberately ambiguous ones. Right direction, but a smaller gap than you'd want if you plan to route on a threshold.
Correction 1: the "wrong type" finding was my mistake. I first wrote that chat models return the declared type unreliably (Gemini answered true when asked for a probability, GPT-5 nano returned strings). Structured output was off. With response_format: { type: "json_schema", strict: true } as the only change, Gemini went from 2/5 to 5/5 and GPT-5 nano from 4/5 to 5/5. So type safety is one line in your request body, not a reason to pick Jev.
Correction 2: run-to-run drift was sampling. At default temperature the chat models changed answers between identical passes (Gemini 26, 25, 26; GPT-5 nano 24, 26, 26). With temperature: 0, every model scored the same on all three passes.
What survived both corrections: latency and cost. On the support-ticket fixture with schemas on, Jev took 386 ms, Gemini 896 ms, Mistral 1,672 ms and GPT-5 nano 7,006 ms.
Limits. One machine, one afternoon, one gateway, five runs a cell; 27 tickets written by me, one task, English only. Enough to retire a general claim, not enough to establish one.
Full tables, per-fixture latency, the vendor's own numbers and a few independent reproductions are here: https://jev-agent.com/benchmarks
To reproduce: python3 -m bench.headtohead --repeats 5 and python3 -m bench.accuracy from https://github.com/WallerChen/jev-measured
I run jev-agent.com, an independent site. It is not affiliated with TypeSafe AI.
Top comments (0)