About a month after Jev's launch, it's clear it opened the eyes of AI developers to a new category: discriminative decision models.
Discriminative models have existed for decades, but Jev arrived with a key difference: it's a purely discriminative foundation model, built for general use through an API. You no longer need to train your system on 1,000 pictures of cats and 1,000 pictures of dogs to get a neural network that reliably tells them apart.
As with every new tool or model, a wave of benchmarks and use cases followed, highlighting the impact these models had on real systems. Reading through them, I noticed a common thread: most of an agent's LLM calls don't need reasoning, they're decisions. Approving a command, picking the next tool, judging whether the task is done, deciding which model to use, and so on. For that you don't need Claude Fable or GPT-6 Sol. You need something cheap, fast, and with structured output.
LangChain integrated Jev into its harness and into LangSmith. OpenAI announced the Decisions API at DevDay. The savings figures ranged from "40x" to "277x" to "444x".
I went after the primary source of each of these numbers, trying to understand what was different in each use case to produce such a wide spread. Spoiler: the thesis survives, but not the way it looks.
Cascading and routing (nothing new)
Using a cheap model and only falling back to an expensive one when needed isn't a new idea, and it comes in two forms.
In cascading, the cheap model answers first and the answer only goes up to the expensive model if it isn't good enough. In other words, the decision about whether a better model is needed comes after the answer. FrugalGPT (Stanford, 2023) showed this can match the performance of the best individual model with up to 98% cost reduction.
In routing, a classifier decides up front which model to call. RouteLLM (LMSYS/Berkeley, 2024) trained routers on human votes about which answer was better and cut costs "by over 2 times in certain cases" without losing quality.
What changed in 2026 is the piece in the middle: instead of a small LLM or a classifier trained for a single decision, we now have general-purpose models that only decide. Any question, any set of answers, no training required.
- Jev (TypeSafe): trained with RLCD (Reinforcement Learning for Calibrated Decisions). It takes a question with a schema of possible answers and returns a choice with a calibrated confidence. TypeSafe claims "40x-200x faster" for queries in this shape, with input at $0.042/MTok.
- Decisions API (OpenAI): announced at DevDay as "real-time decision-making with Decisions API, powered by GPT-6 Luna", to "classify content, route requests, or choose an agent's next action". One slide showed 150 ms versus 1.6 s for regular Luna.
So far, so good. Now for the numbers.
Where each number comes from
"444x" is TypeSafe's own demo
The "444.6x cheaper" comes from a demo on TypeSafe's own homepage, with workflows built by their team. There's no published methodology to evaluate.
"277x" comes from an independent paper
The 277x, on the other hand, isn't TypeSafe marketing. It comes from an independent CMU paper, JEV-as-a-Judge: Accept When Confident, Escalate When Unsure, which measured $0.044 versus $12.182 per thousand judgments compared to GPT-6. But the authors themselves limit its scope: they only evaluated Jev as a judge of finished outputs. Whether it works as well inside an agent, choosing a tool or the next action, "is not tested here."
This paper was the most robust evidence I found, precisely because it measures quality, not just cost: 5,172 judgments comparing Jev with 16 other judges, with blinded human adjudication where the judges disagreed.
| Benchmark | Jev | GPT-6 |
|---|---|---|
| RewardBench (1,500) | 92.5% | 92.5% |
| HaluEval (3,000) | 87.3% | 88.4% |
| JudgeBench (350) | 78.6% | 93.1% |
| Code (1,302) | 78.8% | 91.7% |
| Logic puzzles (98) | 68.4% | 95.9% |
On "recognition" judgments, meaning you can judge just by reading the material given to the judge (for example, which reply is more helpful to a customer, or whether it's supported by the reference document), Jev technically ties (in the authors' words, it comes "within three points of GPT-6 wherever a verdict can be read off the text"). When the verdict has to be derived, meaning the judge has to solve the problem to know whether the answer is right (math, code, logic), it loses 13 to 28 percentage points. The authors say this explicitly.
"83x" or "13%"? Depends on who you compare against
LangChain tested Jev as an agent eval judge. They took 5 responses from a simple weather agent and asked each judge to evaluate each response 100 times (500 judgments in total), to measure not only whether it's right, but whether its verdict is consistent. All of it checked against a human reviewer.
| Judge | Agreement with human | Total cost |
|---|---|---|
| Jev | 100% | $0.34 |
| GPT-5.6 Luna | 96.4% | $0.39 |
| GPT-5.6 Terra | 99.8% | $2.90 |
| Claude Sonnet 4.6 | 80.0% | $28.17 |
Against Sonnet, Jev is roughly 83x cheaper. But against Luna, the direct competitor of a decision model, the gap shrinks to 13%. And the one used in the study is GPT-5.6 Luna, a generation before the GPT-6 Luna that powers the Decisions API. The real gain shown here is something else: variance. Jev's average per-case variance was 92 to 913 times lower than the LLMs'. For a judge that runs on every change to the agent, that matters more than a few cents. If the judge wobbles, you can't tell whether the score dropped because the agent got worse or because the judge changed its mind.
To their credit, the authors call it "a narrow test": just 5 cases with a single weather agent. And "the Jev service version was not available in the experiment metadata."
TypeSafe itself admits its numbers are "on the higher end of real world gains", and that its speedups "tend to be a lot less" against the non-reasoning modes of LLMs, like Luna.
"−64% with no quality loss", with a catch
LangChain's model router in Open SWE, their open source coding agent, cut the median cost per conversation (thread) by 64% ($0.94 versus $2.61) compared to always using the most expensive model, GPT-6 Astra. The mean dropped 42%, and the most expensive 10% of conversations dropped 37%. The A/B test ran on 973 conversations from internal engineers, and quality (the share of conversations that ended in a merged PR) came in at 29.2% versus 27.3%. With p=0.49, the difference could be chance; in other words, the router saved money without hurting quality.
Two details change how you read this:
- The router decides once per thread, classifying only the first human message. It doesn't re-route midway. It picks between three tiers (GLM-5.3-Flash, GPT-5.6 Sol, GPT-6 Astra). It started as an LLM with structured output and moved to Jev, "which made classification almost 50× faster".
- There was a second arm: router versus "always the fast model". It was shut down in less than a day, before reaching statistical significance, because the always-fast arm "was disrupting their productivity due to low output quality".
In short: the cheap decision worked. The cheap execution didn't.
Conclusion
What the evidence supports
- Cascading with escalation. In the CMU paper, Jev answers first and its answer is only accepted when it's confident. The rest goes up to GPT-6. On the 1,610 held-out pairs, Jev handled 69% of decisions, the system came out 0.9 points more accurate than GPT-6 alone, and cost 41% as much. It's not 277x, it's about 2.4x (still a meaningful saving with no loss in quality).
- Route early, once, with a classifier. LangChain's router picks the model on the first message and keeps it until the end of the conversation, without trying to optimize every call. It's simple, avoids the costs of switching models midway, and it was the only case with a production A/B test showing savings without quality loss.
- Consistency, not just cost. In LangChain's test, the saving against Luna was only 13%, but Jev agreed with the human on all 500 judgments and wobbled 92 to 913 times less than the LLMs. For a judge that evaluates the agent on every change, that's worth more than the money saved: the score only changes when the agent changes.
- Inefficiency on decisions that require reasoning. "Is this PR correct?" has a yes-or-no answer, but answering it requires understanding the code. It's derivation disguised as a decision, and it's exactly where Jev loses 13 points. In those cases, either you accept the error, or you escalate to the big model and lose the savings.
Open questions
During this research, I realized the premise I mentioned at the start of this article, that "most of an agent's calls are decisions", has not been measured. I found no study with real agent traces showing what fraction of calls are pure decisions and what fraction are reasoning or generation.
The closest thing I found was NVIDIA's paper Small Language Models are the Future of Agentic AI, which estimates that 40% to 70% of the queries in MetaGPT, Open Operator, and Cradle "could be reliably handled" by specialized SLMs. But that's the authors' estimate, not an actual measurement (and it's worth remembering that NVIDIA sells infrastructure for SLMs).
Another point that matters in real life, and that the studies don't measure, is the downstream cost of a wrong decision. How much does it cost, end to end in your workflow, when a command gets approved that shouldn't have been, or a task gets routed to the wrong tier?
Both of these depend heavily on how you built your solution and your workflow; they're practically unique to each case. But if you have an agent in production, these are probably the numbers that will matter most in your evaluation.
Final thoughts
Decision models are a real and useful harness primitive. But the real gain might look like "2.4x with cascading and escalation" or "−64% routing once per thread", not always "277x". The number you see depends on the baseline someone picked, and a cheap decision in the wrong place, like one that requires reasoning, can end up costing more than the big model.
Be wary of multipliers without context. Ask: against whom, measuring what, with how many cases?
In the end, none of these studies promised a ready-made answer or a recipe to follow. The publications themselves (as we saw above) are quite clear about their limitations. I saw a lot of people ignore all of that context and focus only on the "277x" or the other eye-catching numbers. The value of these studies isn't in the number, it's in showing in which scenario each approach works and which variables change the outcome. The lesson to take from them isn't "which model is best", but "what of this applies to my product".
References
- Chen, Zaharia, Zou. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. Stanford, 2023.
- Ong et al. RouteLLM: Learning to Route LLMs with Preference Data. LMSYS/Berkeley, 2024.
- JEV-as-a-Judge: Accept When Confident, Escalate When Unsure. CMU, 2026.
- Belcak et al. Small Language Models are the Future of Agentic AI. NVIDIA, 2025.
- TypeSafe. Introducing System One Models & Jev.
- LangChain. Can Jev Be a Better Agent Evaluator?.
- LangChain. How to Build a Model Router in the Harness.
- OpenAI Developers. Decisions API announcement at DevDay.
- The Decoder. OpenAI expands Codex and its API at DevDay with security scans, a Decisions API, and Ultrafast
Top comments (0)