Your AI Returned Perfect JSON. It Still Made the Wrong Decision.

What Jev, structured outputs, and calibrated decisions reveal about building reliable AI workflows.

I would consider Jev for decisions in my project where the answer must come from a defined set of options. Its probability estimates interest me, but I would want to check how well they match actual outcomes before relying on them.

This article includes AI-generated drafting and research synthesis, with the author's perspective on possible uses and evaluation. It does not report a hands-on Jev benchmark.

From open-ended generation to bounded decisions. AI-generated illustration.

Consider a hypothetical support workflow. A customer sends this message:

“You charged me twice. I’ve already emailed support, and nobody has replied.”

Your application has three departments:

Billing. Technical support. Sales.

You send the message to an LLM, explain the categories, specify a JSON schema, and wait for it to generate:

{
  "department": "billing"
}

The answer is useful. But the model did not need to invent anything. Every possible answer was already in your application.

That raises an engineering question: when the output is a choice, how much generative machinery do we need?

TypeSafe AI’s Jev is built around that question. It accepts context and typed questions, then returns bounded decisions with probabilities rather than open-ended prose. The company calls it a System One model. TypeSafe documentation

The interesting opportunity is deciding which parts of an AI application actually need generation.

The dropdown is easy. Understanding the message is hard.

A fixed list of answers does not make the task trivial.

Consider these messages:

• “My card was charged twice.”
• “The payment webhook fires twice.”
• “Can your enterprise plan support two billing accounts?”

All mention payments. They may belong to different departments.

A keyword rule could route all three to billing. A useful model needs to interpret intent, distinguish a financial problem from an integration bug, and recognize a purchasing question.

That is where learned judgment helps.

The mistake is assuming learned judgment must always arrive as generated text.

For bounded tasks, the application already knows the output vocabulary. What it needs is a reliable mapping from messy input to those options—and a useful indication of uncertainty.

Similar wording can imply different support destinations. Illustrative examples; actual routes depend on your policy.

What Jev actually returns

Jev’s interface has three main question types:

• Choice: select among defined options.
• Score: evaluate against an ordered rubric.
• Noul: estimate whether a proposition is true.

Choice and Score return probability distributions and a confidence statistic. Noul returns a value between zero and one. Multiple questions can be evaluated independently against the same input. TypeSafe introduction

For our support workflow, those questions might be:

1. Which department should handle the message?
2. How urgent is it under our support policy?
3. Does it describe a possible duplicate charge?

The application then combines the answers with ordinary code.

A message can be billing-related without being urgent. An angry tone does not necessarily mean a service outage. Keeping these judgments separate makes the routing policy easier to inspect and change.

The model interprets the message. Your code owns the consequences.

Choice, Score, and Noul answer different kinds of bounded questions. Illustrative examples based on TypeSafe’s documentation; no API results are shown.

Where RLHF, RLVR, and RLCD fit

These acronyms describe training approaches, not competing categories of chatbot.

RLHF—Reinforcement Learning from Human Feedback—uses human feedback to shape model behavior. In the InstructGPT approach, human demonstrations supported supervised training, and rankings of model responses helped train a reward model used for reinforcement learning. This improved instruction following and user preference ratings. It did not guarantee correctness. InstructGPT paper

RLVR—Reinforcement Learning with Verifiable Rewards—uses rewards that can be checked programmatically. A mathematical answer can be compared with a known result; generated code can be evaluated through tests. The quality and coverage of the verifier still matter. DeepSeek-R1 is a prominent example of reinforcement learning applied to reasoning, including rule-based rewards in its training process. DeepSeek-R1 paper

RLCD—Reinforcement Learning for Calibrated Decisions—is TypeSafe’s name for Jev’s training approach. The stated target is bounded answers accompanied by probabilities that meaningfully reflect uncertainty. TypeSafe AI primer

The distinction is the optimization target:

• Human feedback asks which response people prefer.
• Verifiable rewards ask whether an output passes the available checks.
• Calibrated decisions aim to make probability estimates correspond to observed outcomes.

None of these objectives, by itself, proves that a deployed system is reliable.

RLHF, RLVR, and TypeSafe’s RLCD emphasize different training objectives. This simplified comparison does not establish relative model performance.

An 80% probability should mean something

Suppose a routing model repeatedly assigns an 80% probability to “billing.”

Across a sufficiently large, relevant set of comparable predictions, that category should be correct about 80% of the time if the probabilities are well calibrated.

That is a statement about groups of predictions. It is not a guarantee about one ticket.

TypeSafe explicitly makes this distinction in its calibration explanation. AI primer

There is another subtlety: Jev’s confidence statistic is derived from its probability distribution. It should not automatically be interpreted as the probability that the selected answer is correct. Confidence documentation

For a business, the useful question is operational:

How many tickets can we route automatically while keeping routing errors below an acceptable level?

A model that handles 70% of incoming tickets reliably and escalates the rest may be more valuable than one that confidently routes everything.

For my proposed use, I would distinguish getting a confidence score from validating it. I would compare predicted probabilities with observed outcomes on representative cases. A confident answer is useful only if the evaluation shows when it deserves that confidence.

A valid answer can still be the wrong answer

TypeSafe’s launch announcement makes a striking claim: Jev cannot hallucinate.

The scope of that claim matters.

If the allowed categories are billing, technical support, and sales, a constrained decision interface can prevent an invented fourth category. That solves an output-validity problem.

It does not prove the selected category is correct.

“Billing” is a valid answer for every request under that schema. It is still the wrong destination for a broken webhook.

TypeSafe’s announcement links its claim to schema matching. Readers should distinguish that guarantee from semantic accuracy. Jev launch announcement

Your application therefore needs more than predefined options. It also needs:

• A route for missing or ambiguous information.
• Evaluation against real examples.
• A fallback when the model is uncertain.
• Monitoring after deployment.

A dropdown limits what the model can say. It cannot make every choice true.

An allowed category can still be the wrong route. This is an illustrative failure case, not a measured Jev result.

What about LLM structured outputs?

This is the strongest counterargument to the pitch: LLMs can already return constrained, structured answers.

Structured-output implementations can restrict generated responses to a supported schema. That makes comparisons against an unconstrained chatbot a weak baseline. Schema adherence also does not eliminate mistakes inside the values. OpenAI’s structured-output explanation

A useful evaluation should compare Jev with the strongest practical alternatives:

1. Deterministic rules.
2. A task-specific classifier.
3. A small LLM using structured output.
4. A larger LLM when the task requires it.

Jev should earn its place through measured accuracy, uncertainty quality, latency, and cost.

Cheap decisions can become expensive mistakes

TypeSafe reports substantial speed and cost advantages in its launch evaluations. Its headline figures include approximately 194 times faster and 445 times cheaper on selected workflows.

Those are vendor-reported results. The company notes that the gains may be toward the high end of real-world improvements. Launch announcement

Its workflow evaluation also uses reference answers derived from frontier-model consensus, rather than treating independently labeled real-world outcomes as ground truth. That is useful context when interpreting the scores. Evaluation methodology

Source: TypeSafe workflow evaluations. Vendor-reported comparison against model-consensus labels; not an independent benchmark.

In the chart's cost view, the vertical axis shows mean accuracy and the horizontal axis shows cost per case on a logarithmic scale. Diamonds represent workflows; circles represent standalone prompts. Further left means cheaper; higher means more accurate.

Jev is the lowest-cost configuration shown, but it does not have the highest accuracy. That makes the chart useful for comparing tradeoffs, not declaring a universal winner.

The points average four workflows equally against model-consensus reference labels. This screenshot shows cost, not latency, so it does not by itself substantiate a speedup claim. Chart and methodology

The cost is impressive to me. But for consequential decisions, including those in finance or healthcare, I would need stronger evidence about the errors that matter for that task. This chart alone does not tell me whether Jev meets that standard. Its average score is not an estimate of accuracy in either industry.

Context handling is another area I would test. TypeSafe documents that irrelevant material in Jev 1.13's input can reduce accuracy, and recommends filtering the input to the information needed for the decision. A larger context window alone would not establish better accuracy. Jev's documented limitations

For a future version, I would want evidence of more reliable decisions with relevant context. I would compare the same cases with concise inputs, necessary additional evidence, and irrelevant detail. That would test context sensitivity; it would not assume that context length caused the gap in the vendor chart.

For the support example, the meaningful cost includes more than inference:

Model calls + retries + manual review + the consequences of misrouting.

An inexpensive decision is valuable when the entire workflow becomes less expensive at the quality level you need.

The regression comparison I would run

Before adopting Jev, I would run a regression comparison against small and larger language models on the same classification task. The point would be to see which approach meets the task's accuracy requirements and whether that performance survives changes to the model or workflow.

A proposed evaluation would include:

1. A shared reference set. Use representative cases with independently reviewed labels and explicit decision rules. Include ambiguous inputs and an option to defer when there is not enough information.
2. Comparable baselines. Compare Jev, a small language model, and a larger LLM with the same evidence and allowed answers. Give the language models structured outputs. Include deterministic rules where they apply.
3. Errors by consequence. Report class-level errors and important failure cases alongside overall accuracy. Compare error rates at the same fraction of cases handled automatically, so a model does not appear safer simply because it defers more work.
4. A separate calibration check. Test probability estimates against observed outcomes. Keep model-specific confidence statistics separate; a confidence score of 0.9 need not mean the same thing across systems. If a baseline does not provide comparable probabilities, report that limitation.
5. Context and version tests. Vary relevant evidence and irrelevant detail, record latency and cost including retries, and rerun the fixed cases after model, prompt, or policy changes. Select thresholds on validation data, then assess them on separate test cases.

This is a test plan, not a completed benchmark. Until the comparison is run, I cannot claim that Jev or either language-model baseline is the right choice for my project.

A sensible support architecture could look like this:

1. Rules handle explicit, unambiguous cases.
2. A decision model classifies the remaining messages.
3. The application checks uncertainty and business policy.
4. It routes the case, requests more information, or escalates for human review.

The model proposes a decision; application code controls routing, clarification, and review. Illustrative architecture.

A generative model can enter later, when someone needs a response drafted, a complicated thread summarized, or an explanation written.

Before choosing a model, write down what the output must be.

If the answer can be an arbitrary paragraph, program, or plan, generation is central to the task.

If the answer must be one of eight labels, a score, or a yes/no judgment, evaluate the available decision tools first.

A lower inference bill is attractive. For a consequential decision, I would first need evidence that the workflow makes the right choice often enough—and defers when it should.

---

Research note: The chart is vendor-reported, the support examples are hypothetical, and the proposed comparison has not been run. The author's use-case and evaluation priorities were edited with AI; the detailed test plan is an AI-assisted proposal.

Further reading: Andrew Batutin’s discussion of Jev and GLiFormer, which prompted this article.