DEV Community: Dwayo AI

Which of your LLM calls are really decisions? What we measured with a System One model

Devji — Mon, 26 Oct 2026 08:46:28 +0000

TL;DR: On three public tasks, Jev (a System One decision model) had a median latency of 346 to 372 ms against 1,035 to 1,057 ms for Claude Haiku 5.5 with thinking off. That is about 2.8 to 3 times faster, at about 2.1 to 2.5 times lower list-price cost. It was 4.0 points ahead on 77-class Banking77 and within noise on the other two. We only tested one small chat model, so this says nothing about frontier models.

Are most production LLM calls really decisions?

Often, yes. Between a user's message and the final reply, an agent routes tickets, classifies intent and rates sentiment. Each of these has a small set of valid answers known in advance, and each runs on almost every request.

Teams usually send these calls to the same chat model that writes the reply. It works, but you pay for generation, wait for it, and then parse free text back into a label. We covered how that cost compounds in why LLM inference costs blow up at scale.

A newer class of model is built only for this job. We wanted to know whether it holds up on real tasks, so we measured it instead of repeating its maker's numbers.

What is a System One model?

A System One model makes fast, structured decisions instead of writing text. You give it text or JSON and a typed question, and it returns a typed answer with a probability, not a paragraph. The name borrows from Kahneman's fast and slow thinking: a chat model reasoning through a problem is the slow side, and a System One model is the fast, intuitive side.

There are three question types:

yes/no: how likely is this statement to be true
choice: pick one option from a set you define
score: place the input on an ordered scale

Jev is the first model of this kind we came across. Three facts shape how you would use it:

It is hosted only, with no downloadable weights.
It accepts text only.
It was in early access when we tested it. We ran version jev-1.13.0 in October 2026.

You will see the same claims repeated: answers in 70 to 500 ms, about $0.042 per million input tokens with free output, and a model that "cannot hallucinate". Most coverage repeats the maker's figures, and independent reproductions are scarce; DataCamp's overview says as much. The "cannot hallucinate" claim means the output is always a valid value from your list. It does not mean the value is right. So we measured how often it was right.

What should you ask before moving an LLM call?

Not every LLM call is a decision. Run each call in your pipeline through these four checks:

Check	What to ask
Closed answers	Can you list every valid answer in advance: a set of labels, a scale, or yes/no? If the answer needs free text or an explanation, keep the chat model.
Text input	Is the input text or JSON? The model we tested does not take images, audio or video.
Latency matters	Is a user or another service waiting on the answer, so that saving a second is felt?
Volume	Does the call run on a large share of requests? A saving of a fraction of a cent only matters at volume.

Four yes answers make a call a strong candidate. Three are worth testing. Two or fewer, leave it on the chat model. A yes does not tell you the new model is accurate enough, which is what the next section is for.

Whichever model you pick, keep it behind an interface so you can swap it later; we explain why in avoiding AI vendor lock-in. To run the check across your whole pipeline in one go, there is a free decision-call audit that scores every call and exports a measurement plan.

How did Jev compare on cost, latency and accuracy?

We ran Jev and Claude Haiku 5.5 on the same examples from public datasets: Banking77 (77 customer-support intents, 500 examples), a 10-class subset of it (400 examples) and a content-safety dataset for a safe-or-unsafe check (500 examples). Datasets: Banking77 (PolyAI, CC-BY-4.0) and the Aegis 2.0 content-safety dataset (NVIDIA, CC-BY-4.0).

How we ran it:

Haiku ran with thinking turned off and a 128-token cap. That is not its default setting; with thinking on, some calls used the whole budget before answering.
Latency was measured from our client, network included, with four calls in flight.
Cost is tokens used times each model's list price on 22 October 2026, not an invoice.
We did not tune prompts for either model.

Task	Accuracy: Jev	Accuracy: Haiku	Median latency: Jev	Median latency: Haiku	Cost per 1,000: Jev	Cost per 1,000: Haiku
Banking77, 77 classes	79.0%	75.0%	372 ms	1,057 ms	$0.0429	$0.1073
Banking77, 10 classes	95.5%	95.8%	367 ms	1,035 ms	$0.0160	$0.0351
Content safety	81.8%	79.0%	346 ms	1,037 ms	$0.0144	$0.0304

Speed and cost. Jev's median latency was 346 to 372 ms against 1,035 to 1,057 ms for Haiku, about 2.8 to 3 times faster. At list prices it cost about 2.1 to 2.5 times less. Its slow tail was modest, with p95 between 463 and 677 ms, though on the safety task that went past the 500 ms top of the range its maker quotes.

Accuracy. On the 77-class task Jev was ahead by 4.0 points (95% interval +1.4 to +7.0, exact McNemar test p = 0.011). There were 38 examples only Jev got right and 18 only Haiku got right. The result still holds after a simple correction for testing three tasks. On the 10-class task the gap was within noise (-0.3 points, p = 1.0), and on the safety task it was too (+2.8 points, interval -0.0 to +5.8, p = 0.08). Each task was one run on one date, so treat the 77-class gap as promising, not settled. On a repeat of identical inputs Jev's accuracy moved slightly, so the last digit of any figure here is noise.

The summary: on these three tasks a decision model was much faster and cheaper than a small chat model, and at least as accurate. That says nothing about frontier models, which we did not test.

Reproduce it. The harness, the pinned model versions and every run record are open source in system-one-decision-bench (Apache-2.0). Rerun it with your own API keys and check our numbers.

What else did the benchmark show?

Accuracy falls as the label set grows. Jev scored 95.5% on 10 classes and 79.0% on 77. Haiku fell further, from 95.8% to 75.0%. The two tasks use different example sets and the 10 classes were hand-picked, so treat this as a rule of thumb, not a controlled test.
Some failures are about the labels, not the model. The classes Jev got most wrong were the same ones Haiku got most wrong. Both models missed order_physical_card in 9 or 10 of 10 examples, usually answering get_physical_card, a label that means almost the same thing. Both also failed on 7 of 9 examples of top_up_by_bank_transfer_charge, answering transfer_fee_charged. When two labels overlap that much, no classifier can separate them. Fix the label set before you blame the model.
Probabilities are not certainty. Of Jev's 500 answers on the 77-class task, 338 landed in the top confidence band (0.9 to 1.0). Average stated confidence there was 98.4%, and actual accuracy was 93.8%. Against Haiku, Jev's calibration was better on two tasks and worse on one, so we do not claim either model is better calibrated.
It does not write or explain. It returns a value and a probability. If you need a stated reason for a decision, for an audit for example, you need another way to get one.

Does a chat-model fallback help?

Barely, in our test. A popular design is a cascade: let the fast model decide, and send the call to a bigger model only when its probability falls below a threshold. We tested it with Jev first and Haiku as the fallback, sweeping the threshold from 0.50 to 0.99.

At the thresholds we would actually pick (the cheapest that kept accuracy within a point of the best), only 5.4%, 0.2% and 0% of calls were escalated on the three tasks. The one gain was on the 77-class task: +1.4 points over Jev alone (interval +0.4 to +2.6) at about 1.14 times Jev's cost. We picked that threshold on the same 500 examples we measured on, so the real gain is probably smaller.

Raising the threshold made things worse. On the 77-class task accuracy fell from 80.4% at a threshold of 0.50 to 75.8% at 0.99, close to Haiku's 75.0%. The reason is simple: the fallback was no more accurate than Jev. A cascade only pays when the second model is stronger than the first, and we only tested a weaker one. So this result says nothing about a cascade with a stronger fallback.

What didn't the benchmark test?

Frontier chat models. We compared against one small model only.
A fine-tuned classifier. For a task like routing, a small classifier trained on your own labelled data is the obvious rival, and we did not test one. Open-source decision models also exist.
Other access routes. We called the model directly over its HTTP API, not through a gateway.
Prompt tuning, other languages and long inputs. One prompt per task, short English text.
Anything over time. One run per task, on one date, on one model version. A hosted model can change under you.

Should you test it on your own data first?

Yes. List your LLM calls and run the four checks. Pick the highest-volume call that passes, then try it on a few hundred of your own labelled examples before you change anything. Measure accuracy, median and p95 latency, and cost per 1,000 decisions on your data, not ours.

Keep the model behind an interface so you can swap it, pin the exact model version in config, and look first at your labels: if your own team would disagree about the right answer, no model will fix that. For the code, see the integration guide. The benchmark harness is a working example of the interface pattern.

Originally published at dwayo.ai. We're Dwayo AI, an AI engineering partner.

We Almost Deployed an AI Debugging Agent. Then We Found a Critical CVE.

Devji — Tue, 20 Oct 2026 06:09:22 +0000

When our client's team evaluated Sentry Seer, the appeal was clear: it could automatically triage production errors, propose fixes, and open pull requests. For a small engineering team, that promised meaningful time savings. We decided not to proceed with the rollout after a serious security vulnerability was disclosed just days before launch.

The vulnerability wasn't a misconfiguration

On September 16, 2026, CERT/CC published VU#212479, describing CVE-2026-90999, a critical trust-boundary violation in Sentry Seer.

Here's the part that matters: Sentry projects commonly expose a public DSN in client-side code so browsers can submit error telemetry. Public DSNs are normal in browser applications because they only accept event submissions, so we hadn't considered ours a significant risk.

CVE-2026-90999 changed that assumption. An unauthenticated attacker can submit a fabricated Sentry event through the public DSN. If Seer is configured to automatically hand issues to a coding agent, that attacker-controlled telemetry flows through Seer's analysis, gets embedded into the agent's prompt as legitimate context, and eventually causes the agent to download and execute attacker-controlled code.

An attacker would not need access to the organization's Sentry account, repository, or infrastructure. Just the DSN already shipped in your frontend JavaScript.

The pattern behind the CVE

This wasn't the first warning. In June 2026, Tenet Security disclosed "agentjacking": an attacker sends a forged error event through a public Sentry DSN, and when a developer later asks their coding agent to look at open Sentry issues, the agent reads the event through Sentry's MCP server and follows the attacker's instructions. Sentry acknowledged the report but declined to fix the root cause, calling it "technically not defensible" at the platform level. CVE-2026-90999 is the same weakness on a different path. Agentjacking needed a developer to prompt the agent. Seer's automated handoff removes that step, so forged telemetry can reach a privileged coding agent with no human involved at all.

It would be easy to file this under "vendor security incident" and move on. But this is structural, and applicable to any AI agent that consumes external telemetry and takes automated action.

Most security reviews focus on the model and the endpoint. Once an AI system can take action, the path from incoming data to execution becomes another part of the system that needs a security review.

In this case, that pipeline had four stages:

Telemetry ingestion — a public DSN accepted events from anyone
AI processing — Seer generated a root-cause analysis
Agent handoff — the analysis was embedded into a prompt sent to a coding agent
Execution — the agent ran code in a privileged environment with source repo access

Each stage assumed the previous one had validated its input. None enforced that assumption end-to-end. The problem was not isolated to one component. It came from the way untrusted data moved through the entire workflow.

The CERT/CC advisory describes it plainly: fabricated events traverse multiple trust boundaries, transforming from untrusted telemetry into instructions the privileged agent executes.

Why this matters for every enterprise evaluating AI agents

Seer isn't unusual in its architecture. Telemetry in, AI reasoning, automated action out: this is becoming the default shape of AI tooling. Error triage, log analysis, incident response, code review. All of it is moving toward agentic pipelines that don't just suggest but execute.

That changes the security question. We realized that evaluating the model alone was insufficient. We also needed to understand which inputs could reach the agent and what permissions the agent had.

Most teams evaluate model accuracy and integration surface. But they do not evaluate the full chain of custody for telemetry.

A framework we now use

We built a short checklist for any AI agent that takes automated action:

Check	What to ask
Inputs	Are all inputs authenticated? Can an external actor trigger the pipeline without credentials? Public endpoints such as DSNs, webhooks, and forms should be treated as untrusted entry points.
Privileges	Does the agent execute code? In what environment? With what scope? If the answer is "our infrastructure, with production access," the bar for input validation is much higher.
Handoffs	Every stage between input and execution is a trust boundary. Are those boundaries enforced, or assumed?
Triggerability	If an unauthenticated user can trigger the workflow, every stage of that workflow needs to be included in the threat model.
Blast radius	If malicious input reaches the agent, what can it do? CERT/CC notes that successful exploitation gives access to connected source repositories, and allows actions permitted by the coding agent's credentials.

The questions are easier to answer before launch than after. Adversarial testing of what an agent does with hostile input is part of our AI agent testing and evaluation work, and the output-schema, whitelist and human-approval pattern is the one we describe in our agent guardrails case study.

What to take away

Agents that triage, diagnose, and fix production issues are too valuable to dismiss because of one CVE.

But this changes how we evaluate them. Our reviews should cover not only whether an agent works, but also how it handles untrusted input before taking action and which team is responsible for those controls.

As of this writing, no vendor patch is available. The mitigations are operational: disable autonomous handoff, sandbox the agent, restrict package installation, treat telemetry as untrusted input.

For enterprises deploying agentic AI in engineering workflows, we now consider these controls a basic requirement for deploying autonomous agents. These tools can still be valuable, but they introduce risks that need to be addressed before deployment.

Originally published on the Dwayo blog. Evaluating an AI agent for your team? Book a 30-minute review.