Which of your LLM calls are really decisions? What we measured with a System One model

TL;DR: On three public tasks, Jev (a System One decision model) had a median latency of 346 to 372 ms against 1,035 to 1,057 ms for Claude Haiku 5.5 with thinking off. That is about 2.8 to 3 times faster, at about 2.1 to 2.5 times lower list-price cost. It was 4.0 points ahead on 77-class Banking77 and within noise on the other two. We only tested one small chat model, so this says nothing about frontier models.

Are most production LLM calls really decisions?

Often, yes. Between a user's message and the final reply, an agent routes tickets, classifies intent and rates sentiment. Each of these has a small set of valid answers known in advance, and each runs on almost every request.

Teams usually send these calls to the same chat model that writes the reply. It works, but you pay for generation, wait for it, and then parse free text back into a label. We covered how that cost compounds in why LLM inference costs blow up at scale.

A newer class of model is built only for this job. We wanted to know whether it holds up on real tasks, so we measured it instead of repeating its maker's numbers.

What is a System One model?

A System One model makes fast, structured decisions instead of writing text. You give it text or JSON and a typed question, and it returns a typed answer with a probability, not a paragraph. The name borrows from Kahneman's fast and slow thinking: a chat model reasoning through a problem is the slow side, and a System One model is the fast, intuitive side.

There are three question types:

• yes/no: how likely is this statement to be true
• choice: pick one option from a set you define
• score: place the input on an ordered scale

Jev is the first model of this kind we came across. Three facts shape how you would use it:

• It is hosted only, with no downloadable weights.
• It accepts text only.
• It was in early access when we tested it. We ran version jev-1.13.0 in October 2026.

You will see the same claims repeated: answers in 70 to 500 ms, about $0.042 per million input tokens with free output, and a model that "cannot hallucinate". Most coverage repeats the maker's figures, and independent reproductions are scarce; DataCamp's overview says as much. The "cannot hallucinate" claim means the output is always a valid value from your list. It does not mean the value is right. So we measured how often it was right.

What should you ask before moving an LLM call?

Not every LLM call is a decision. Run each call in your pipeline through these four checks:

Check	What to ask
Closed answers	Can you list every valid answer in advance: a set of labels, a scale, or yes/no? If the answer needs free text or an explanation, keep the chat model.
Text input	Is the input text or JSON? The model we tested does not take images, audio or video.
Latency matters	Is a user or another service waiting on the answer, so that saving a second is felt?
Volume	Does the call run on a large share of requests? A saving of a fraction of a cent only matters at volume.

Four yes answers make a call a strong candidate. Three are worth testing. Two or fewer, leave it on the chat model. A yes does not tell you the new model is accurate enough, which is what the next section is for.

Whichever model you pick, keep it behind an interface so you can swap it later; we explain why in avoiding AI vendor lock-in. To run the check across your whole pipeline in one go, there is a free decision-call audit that scores every call and exports a measurement plan.

How did Jev compare on cost, latency and accuracy?

We ran Jev and Claude Haiku 5.5 on the same examples from public datasets: Banking77 (77 customer-support intents, 500 examples), a 10-class subset of it (400 examples) and a content-safety dataset for a safe-or-unsafe check (500 examples). Datasets: Banking77 (PolyAI, CC-BY-4.0) and the Aegis 2.0 content-safety dataset (NVIDIA, CC-BY-4.0).

How we ran it:

• Haiku ran with thinking turned off and a 128-token cap. That is not its default setting; with thinking on, some calls used the whole budget before answering.
• Latency was measured from our client, network included, with four calls in flight.
• Cost is tokens used times each model's list price on 22 October 2026, not an invoice.
• We did not tune prompts for either model.

Task	Accuracy: Jev	Accuracy: Haiku	Median latency: Jev	Median latency: Haiku	Cost per 1,000: Jev	Cost per 1,000: Haiku
Banking77, 77 classes	79.0%	75.0%	372 ms	1,057 ms	$0.0429	$0.1073
Banking77, 10 classes	95.5%	95.8%	367 ms	1,035 ms	$0.0160	$0.0351
Content safety	81.8%	79.0%	346 ms	1,037 ms	$0.0144	$0.0304

Speed and cost. Jev's median latency was 346 to 372 ms against 1,035 to 1,057 ms for Haiku, about 2.8 to 3 times faster. At list prices it cost about 2.1 to 2.5 times less. Its slow tail was modest, with p95 between 463 and 677 ms, though on the safety task that went past the 500 ms top of the range its maker quotes.

Accuracy. On the 77-class task Jev was ahead by 4.0 points (95% interval +1.4 to +7.0, exact McNemar test p = 0.011). There were 38 examples only Jev got right and 18 only Haiku got right. The result still holds after a simple correction for testing three tasks. On the 10-class task the gap was within noise (-0.3 points, p = 1.0), and on the safety task it was too (+2.8 points, interval -0.0 to +5.8, p = 0.08). Each task was one run on one date, so treat the 77-class gap as promising, not settled. On a repeat of identical inputs Jev's accuracy moved slightly, so the last digit of any figure here is noise.

The summary: on these three tasks a decision model was much faster and cheaper than a small chat model, and at least as accurate. That says nothing about frontier models, which we did not test.

Reproduce it. The harness, the pinned model versions and every run record are open source in system-one-decision-bench (Apache-2.0). Rerun it with your own API keys and check our numbers.

What else did the benchmark show?

• Accuracy falls as the label set grows. Jev scored 95.5% on 10 classes and 79.0% on 77. Haiku fell further, from 95.8% to 75.0%. The two tasks use different example sets and the 10 classes were hand-picked, so treat this as a rule of thumb, not a controlled test.
• Some failures are about the labels, not the model. The classes Jev got most wrong were the same ones Haiku got most wrong. Both models missed order_physical_card in 9 or 10 of 10 examples, usually answering get_physical_card, a label that means almost the same thing. Both also failed on 7 of 9 examples of top_up_by_bank_transfer_charge, answering transfer_fee_charged. When two labels overlap that much, no classifier can separate them. Fix the label set before you blame the model.
• Probabilities are not certainty. Of Jev's 500 answers on the 77-class task, 338 landed in the top confidence band (0.9 to 1.0). Average stated confidence there was 98.4%, and actual accuracy was 93.8%. Against Haiku, Jev's calibration was better on two tasks and worse on one, so we do not claim either model is better calibrated.
• It does not write or explain. It returns a value and a probability. If you need a stated reason for a decision, for an audit for example, you need another way to get one.

Does a chat-model fallback help?

Barely, in our test. A popular design is a cascade: let the fast model decide, and send the call to a bigger model only when its probability falls below a threshold. We tested it with Jev first and Haiku as the fallback, sweeping the threshold from 0.50 to 0.99.

At the thresholds we would actually pick (the cheapest that kept accuracy within a point of the best), only 5.4%, 0.2% and 0% of calls were escalated on the three tasks. The one gain was on the 77-class task: +1.4 points over Jev alone (interval +0.4 to +2.6) at about 1.14 times Jev's cost. We picked that threshold on the same 500 examples we measured on, so the real gain is probably smaller.

Raising the threshold made things worse. On the 77-class task accuracy fell from 80.4% at a threshold of 0.50 to 75.8% at 0.99, close to Haiku's 75.0%. The reason is simple: the fallback was no more accurate than Jev. A cascade only pays when the second model is stronger than the first, and we only tested a weaker one. So this result says nothing about a cascade with a stronger fallback.

What didn't the benchmark test?

• Frontier chat models. We compared against one small model only.
• A fine-tuned classifier. For a task like routing, a small classifier trained on your own labelled data is the obvious rival, and we did not test one. Open-source decision models also exist.
• Other access routes. We called the model directly over its HTTP API, not through a gateway.
• Prompt tuning, other languages and long inputs. One prompt per task, short English text.
• Anything over time. One run per task, on one date, on one model version. A hosted model can change under you.

Should you test it on your own data first?

Yes. List your LLM calls and run the four checks. Pick the highest-volume call that passes, then try it on a few hundred of your own labelled examples before you change anything. Measure accuracy, median and p95 latency, and cost per 1,000 decisions on your data, not ours.

Keep the model behind an interface so you can swap it, pin the exact model version in config, and look first at your labels: if your own team would disagree about the right answer, no model will fix that. For the code, see the integration guide. The benchmark harness is a working example of the interface pattern.

Originally published at dwayo.ai. We're Dwayo AI, an AI engineering partner.