TypeSafe launched Jev on September 15 and it reached OpenRouter three days later. Since then nearly every article about it has repeated the two numbers from TypeSafe's home page: 193.6x faster, 444.6x cheaper.
We wanted to evaluate it for our use cases, so we ran an experiment with 100 support tickets. Each ticket had four types of questions, resulting in 400 decisions per model. Jev against Claude Sonnet 5, GPT-5.6 Sol and Gemini 3.8 Flash. Same tickets, same order, same wording, 20 requests in flight for every lane.
On that workload Jev came out 4x to 7x faster and 31x to 65x cheaper.
Both numbers are honest
TypeSafe's launch post footnotes the headline figures: "we expect that these are on the higher end of real world gains." that implies their claim is against frontier models. Most articles quoting the 193.6x left that out.
Their ratio has two sides, and only one moved when we measured it.
Jev's side checks out. They state a response time of 70ms to 500ms end to end. We measured a 474 ms median across 100 tickets.
The other side is the whole gap. They compared against GPT-6 Astra and Fable 5.1 on multi-step workflows, clocking those at 3 to 329 seconds. We compared against mid-tier models on single triage calls, clocking those at 1.9 to 3.5 seconds. Same numerator, a much smaller denominator, a much smaller multiple. Which are much closer to real use.
A multiple is a property of a comparison, not of a model. Theirs is a best case under controlled scenario and they say so. Ours is close to a floor. Therefore, your number depends on what you are comparing against.
What we measured
Jev clears the queue in 8 seconds. Gemini is still on ticket 20. Zero failures and zero retries in all four lanes, so this is not a reliability story.
Three tenths of a cent against twenty. At 10,000 tickets a month that is about $3.69 a year against $239. Nobody goes bankrupt either way, which is why this line item never gets revisited. The mechanism is just tokens: text models pay to think and to answer, whereas, Jev's output token is free.
Latency is the one you cannot absorb, because it sits between the customer pressing send and anything happening. At 474 ms routing runs inline. At 3.4 seconds it becomes a background job, which means a queue, retries and a dashboard for when it backs up. Watch the tail too: Gemini's slowest request took 18.4 seconds.
Where our benchmark broke
The obvious next question is which model is better at understanding intent. Answering it was harder than everything above combined.
Every ticket was written backwards from a hidden label i.e, owning team, urgency, money-back request, hostile tone were sampled at random and handed to a separate model to dramatize. That label is one answer key. The majority answer of the other three models, each model's own vote withheld, is the second.
Refund and angry hold at 94 to 99 percent under both gradings. Team and urgency collapse against the labels, for all four models, built by four different companies, on exactly the same two questions.
When that happens, the answer key is what is wrong. On urgency the labels put every model near 47 percent, about what guessing gets you on a four level scale, while the models agree with each other 70 to 82 percent of the time. They are not confused. The label is.
If you evaluate a triage system against labels you generated, you are probably measuring your label generator.
This is also the fair reading of the criticism aimed at TypeSafe, whose evals grade by agreement with other frontier models rather than ground truth. That is a real weakness. It is also more understandable than it looks, because we tried the alternative and our ground truth came out worse. Our consensus grading has the same hole, and we will say it plainly: it measures conformity, not truth.
Any two models give identical answers on all four fields for 50 to 72 percent of tickets. On roughly a third of your queue, changing the model changes the answer.
The number that actually runs a helpdesk
You are never automating everything, so accuracy is the wrong target. The useful question is whether the system knows when it does not know.
Every ticket comes with a confidence score, and the score is honest. When Jev says it is sure, it is right. When it hedges, it is genuinely shaky.
47 of the 100 tickets came back at 0.99 or higher, and all 47 matched what the other three models said. On the 18 tickets Jev was least sure about, agreement fell to 72 percent. It flagged its own weak cases.
That gives you a rule: auto-route above 0.90, send the rest to a person. Here that clears 61 tickets (14+47) at 98.4 percent agreement and puts 39 (18+21) on a human desk with the likely teams and odds attached, not a blank queue. The 0.90 is a dial, raise it to automate less and miss less.
This matters more than any speed or cost number above. You do not need a model that is right 99 percent of the time to take triage off a support team. You need one that knows which answers to trust, and a cutoff you picked on purpose. That is the shape our custom AI agent and helpdesk builds keep landing on: automate the confident majority, route every exception to a person with the reasoning attached.
Caveats
- One workload. Single-call support triage, not the multi-step workflows TypeSafe's figures are based on.
- Mid-tier competitor. Against GPT-6 Astra and Fable 5.1, our multiples would be much larger.
- One run. Latency moved by seconds between identical runs. These figures are recorded, not deterministic.
- 100 synthetic tickets. Enough to see a 65x gap. Not enough to separate models two points apart.
- Consensus grading measures conformity, not truth.
- Batching is untested. 20 tickets per call would cut the text lanes' cost, and would reintroduce the JSON reliability problems that forced this dataset to be built one item per call. That is the strongest open objection to our setup.
Check our work
The recorded run, the exact request bodies, all 100 tickets and every model's answer are published as JSON. The report page replays that file rather than calling any API, and a validation script asserts the replay lands within 5 percent of the measured wall clock.
See the full interactive report and raw data
If we have misread TypeSafe's methodology, or your run disagrees with ours, say so. The data is published so it can be argued with.
Working on support automation? BrillMark builds custom AI agents and helpdesks around your own knowledge base, on your infrastructure, with your model keys and your escalation rules. Get a free working demo built on your content.
Top comments (0)