This is a submission for the Kaggle Benchmarking Challenge.
Jev Is Not Trying to Be Another Chatbot — It Is Trying to Become a Decision Layer for Software
AI has spent the last few years getting dramatically better at generating things.
Code. Emails. Images. Plans. Explanations. Entire applications.
But software does something else thousands—or millions—of times per day:
It makes tiny decisions.
Should this request go to billing or support?
Should this document be reviewed?
Should this agent retry?
Which tool should run next?
Is this input relevant enough to keep?
Should a human take over?
For many of those decisions, we have been using a very expensive tool: a general-purpose language model.
Jev takes the opposite approach.
Instead of asking an AI to write a paragraph that our software has to interpret, Jev is designed to return a structured decision that software can use directly.
That sounds like a small API design choice.
I think it is a much bigger architectural idea.
🧠 The core idea
The easiest way to understand Jev is:
┌──────────────────┐
│ STATE │
│ │
│ text / JSON │
│ application data │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ JEV │
│ │
│ Choice / Score │
│ / Noul │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ TYPED DECISION │
│ + probabilities │
│ + confidence │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ YOUR PROGRAM │
│ │
│ route / filter │
│ rank / branch │
│ escalate / act │
└──────────────────┘
A normal LLM workflow often looks more like:
State
↓
LLM
↓
Generated text
↓
Parser
↓
Validation
↓
Retries / error handling
↓
Application logic
Jev is designed to remove the generated-text middleman for bounded decisions.
TypeSafe describes Jev as its first System One model: a model class intended for fast, structured decisions inside software rather than free-form text generation.
LLMs generate strings. Jev returns decisions.
That distinction is the entire story.
What is a "System One" model?
The name is inspired by the familiar distinction between fast, intuitive judgment and slower, deliberative reasoning.
But in software, the useful interpretation is simpler:
Use a fast intelligence primitive for the small judgments inside a larger software system.
The model is not supposed to replace your entire application.
It sits inside your application.
For example:
User message
│
▼
┌───────────────┐
│ Jev │
│ "What kind of │
│ request is │
│ this?" │
└───────┬───────┘
│
┌────┴────┐
▼ ▼
Billing Technical
│ │
▼ ▼
Code Tool
That is fundamentally different from:
"Hey AI, please think through this request and tell me what
my application should probably do next..."
The first interface is a decision API.
The second is a conversation with a language model.
The three primitives
Jev currently exposes three important question types.
| Primitive | What it does | Typical use |
|---|---|---|
Choice |
Picks one option | Routing, classification, tool selection |
Score |
Scores something against a rubric | Quality, urgency, risk, relevance |
Noul |
Estimates whether a statement is true | Verification, gating, yes/no decisions |
The interesting part is that these are not three ways of asking a chatbot a question.
They are three typed interfaces for software decisions.
1. Choice — "Which one?"
Imagine an incoming support ticket:
"My payment went through twice and I still cannot access my subscription."
Your application might need:
billing
technical
account
With an LLM, you might ask:
Classify this support ticket into one of:
billing, technical, account.
Return JSON.
Then your application has to trust the format, parse it, validate it, and handle whatever weird edge case the model produces.
With Jev, the question itself is structured:
from typesafe_sdk import Choice, TypeSafeClient
client = TypeSafeClient()
response = client.system_one(
state="My payment went through twice and I still cannot access my subscription.",
questions={
"department": Choice(
instructions="Which team should handle this request?",
criteria={
"billing": "Charges, refunds, invoices, payment problems",
"technical": "Bugs, errors, broken integrations",
"account": "Login, permissions, account access",
},
),
},
)
print(response.answers["department"].choice)
The important output is not a paragraph.
It is a value your code can branch on.
if response.answers["department"].choice == "billing":
route_to_billing()
That is much closer to calling a function than chatting with an AI.
2. Score — "How much?"
Some decisions are not categories.
They are measurements.
For example:
How urgent is this support message?
You might define:
0 → Not urgent
1 → Somewhat urgent
2 → Critical
or:
0 → Low quality
1 → Acceptable
2 → Excellent
The important thing is that your code defines the rubric.
That makes the model a judgment component rather than the owner of the final policy.
from typesafe_sdk import Score
quality = Score(
instructions="How useful is this answer to a developer?",
criteria=[
"Incorrect or unusable",
"Partially useful",
"Correct and useful",
],
)
Now the model estimates where the state falls on your predefined scale.
Your application decides what to do with the result.
3. Noul — "Is this true?"
This is the simplest primitive.
You give Jev a statement such as:
"This ticket contains a request for a refund."
and ask:
from typesafe_sdk import Noul
refund_request = Noul(
instructions="Does this message contain a request for a refund?"
)
The result is a probability-like value from 0 to 1.
That opens up an interesting class of software:
if refund_request.noul > 0.9:
start_refund_review()
or:
if spam_score > 0.8 and refund_request.noul < 0.1:
auto_route()
Suddenly, AI becomes part of ordinary application logic.
🤯 Why this is interesting
Here's the mental shift I find most important.
We normally think about AI as a thing that produces content.
Jev suggests thinking about AI as a thing that produces uncertainty-aware decisions.
That is a much smaller interface.
And smaller interfaces are powerful.
Compare:
Generate a response explaining which tool should be called.
with:
Choose:
[search_docs, execute_sql, browse_web, ask_human]
The second one gives the model much less freedom.
But that constraint is the feature.
The application remains in control.
"But can't normal LLMs already return JSON?"
Yes.
And this is where the distinction becomes important.
A modern LLM can be instructed to produce:
{
"action": "search_docs",
"confidence": 0.91
}
That is useful.
But the model is fundamentally still a text-generation system whose output has been constrained or structured for a downstream consumer.
Jev is designed around the decision interface from the beginning.
The conceptual difference is:
LLM:
question
↓
tokens
↓
structured text
↓
parser
↓
application
versus:
Jev:
state + typed question
↓
typed decision
↓
application
That sounds subtle.
Inside a production system, it can be a very important distinction.
⚡ Why the speed matters
The other reason Jev is interesting is not just its output format.
It is the economics of putting intelligence inside loops.
TypeSafe currently publishes Jev at $0.042 per million input tokens and describes typical end-to-end response times of roughly 70–500 ms for its System One workloads.
That creates an interesting possibility:
What if AI decisions became cheap enough to use like infrastructure?
Imagine an agent processing:
10 documents
×
8 classification checks
×
5 routing decisions
The AI layer quickly becomes a large number of tiny calls.
With a conventional generative model, you are paying for text generation at each stage.
With a decision model, the target workload is much narrower.
🏗️ The architecture I find most interesting
I don't think Jev should replace LLMs.
I think it makes more sense as a second layer.
Something like this:
┌──────────────────────┐
│ APPLICATION │
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ JEV │
│ │
│ classify │
│ score │
│ verify │
│ route │
│ gate │
└───────┬───────┬──────┘
│ │
confident uncertain
│ │
▼ ▼
execute ┌───────────┐
│ LLM / │
│ human │
└───────────┘
This is where the idea gets really powerful.
Cheap decisions first.
Expensive reasoning only when necessary.
🔀 A hybrid AI agent
Suppose an autonomous coding agent receives an issue.
It could work like this:
Issue arrives
│
▼
Jev: Is this actionable?
│
┌───┴────┐
│ │
No Yes
│ │
▼ ▼
Ask Jev: What type?
│
┌───┼─────────┐
▼ ▼ ▼
bug feature question
│ │ │
└─────┴─────────┘
│
▼
Use appropriate tool
│
▼
LLM for deep work
│
▼
Jev: verify
│
┌─────┴─────┐
▼ ▼
pass uncertain
│ │
▼ ▼
ship human review
Notice what happened.
The LLM did not disappear.
It became one component instead of the entire control plane.
🧪 A real benchmark question
This is the part I wanted to explore for the Kaggle Benchmarking Challenge.
Instead of asking:
"Which model writes the best answer?"
I want to ask:
"Which model makes the best decision when the correct action is not always to act?"
That produces a very different benchmark.
For example, give every model the same software scenario:
A pull request changes authentication middleware.
The tests pass.
The PR description says the change is required for a new API.
However, the repository contains an older mobile client
that still uses the previous authentication flow.
The possible decisions could be:
| Decision | Meaning |
|---|---|
SHIP |
Safe to merge |
CHANGE |
Modify before merging |
ASK |
More information is required |
STOP |
Known unacceptable risk |
Now we can test something that matters in real automation:
Can the model distinguish "I can answer this" from "I should act on this"?
That's a much more interesting failure mode than whether an LLM can produce valid JSON.
📊 What I would measure
A useful benchmark should not collapse everything into one accuracy number.
I would measure at least:
| Metric | Question |
|---|---|
| Decision accuracy | Was the selected action correct? |
| Calibration | Does confidence track correctness? |
| Abstention quality | Does the model ask when it should? |
| False-action rate | How often does it act when it should not? |
| False-stop rate | How often does it block something safe? |
| Latency | How quickly can the decision be made? |
| Cost | How much does each decision cost? |
| Consistency | Does rephrasing change the result unnecessarily? |
That creates a much more useful picture of an AI system.
🔬 There is already evidence worth looking at
Independent public benchmark work is already starting to test Jev in this direction.
For example, a published JevBench run evaluated 242 typed decisions per system. In that run, Jev 1.13.0 reported:
| Metric | Jev 1.13.0 |
|---|---|
| Accuracy | 96.3% |
| Cost / 1K decisions | $0.027 |
| p50 latency | 0.65 s |
| p95 latency | 0.72 s |
| ECE | 0.027 |
| Valid answers | 100% |
| Exact-sum answers | 100% |
These are measurements from the benchmark authors' published run—not numbers I independently reproduced.
The interesting part is not "Jev won."
The interesting part is that the benchmark is measuring different axes at the same time.
For decision systems, accuracy without calibration is incomplete.
Speed without correctness is useless.
Cheap but unreliable automation can be expensive in disguise.
🧩 The hidden superpower: probabilities
This may be the most important architectural idea.
Suppose your system needs to choose between:
A = 0.51
B = 0.49
A naive application sees:
A
A probability-aware application sees:
A, but barely.
That is a huge difference.
Your code can define:
if confidence >= 0.90:
automate()
else:
escalate()
Or:
if confidence >= 0.95:
action = "automatic"
elif confidence >= 0.70:
action = "review_queue"
else:
action = "human"
The model produces the uncertainty.
The application owns the policy.
That separation is healthy software architecture.
🧠 And this changes how prompts should be written
Traditional prompting often looks like:
Analyze this problem deeply and provide the best answer.
A System One-style question should look more like:
Choose the deployment state.
Options:
- safe
- needs_review
- unsafe
Or:
Score the relevance of this document from 0 to 3.
Or:
Does this message contain a security incident?
The question is:
small
atomic
specific
machine-actionable
TypeSafe's documentation explicitly recommends decomposing complex judgments into smaller questions and combining those results in code.
That is a very software-engineering-heavy way of using AI.
🔥 This is where Jev gets weirdly interesting
Imagine processing one document.
Instead of asking one giant prompt:
Analyze this document and give me a complete structured report.
you ask:
Is it relevant?
↓
How urgent is it?
↓
Which department owns it?
↓
Does it contain sensitive information?
↓
Does it require human review?
Those are separate questions.
And they can be combined into an ordinary program:
if sensitive and urgency >= 2:
escalate()
elif relevance < 0.2:
discard()
elif department == "billing":
route("billing")
This is almost like adding a fuzzy intelligence layer to if statements.
🧱 The big limitation
There is an important catch.
Jev is not a replacement for a generative model.
It is not designed to:
- write an article
- implement a feature
- generate a long explanation
- produce arbitrary source code
- replace a reasoning-heavy agent
That's not a bug.
It is the boundary of the product.
A useful architecture therefore looks like:
Code
│
├── deterministic rules
│
├── Jev
│ ├── classify
│ ├── score
│ ├── verify
│ └── route
│
└── LLM
├── reason
├── generate
├── code
└── explain
Use each tool for what it is good at.
⚠️ "Zero hallucinations" needs a careful interpretation
TypeSafe talks about Jev being unable to hallucinate in the traditional text-generation sense.
That's understandable because Jev does not generate arbitrary prose.
But this should not be interpreted as:
"Jev cannot be wrong."
Those are different things.
A model can return:
{
"choice": "billing",
"confidence": 0.98
}
and still be incorrect.
Type-safe output guarantees the shape of the answer.
It does not magically guarantee the truth of the decision.
That distinction matters enormously when putting AI inside production systems.
🧪 My benchmark idea: "Should the Agent Act?"
For the Kaggle challenge, I would turn this into a benchmark around actionability under uncertainty.
Each task would contain:
- State — the information currently available to the agent.
- Decision — the action the system could take.
- Hidden risk — a detail that makes some actions unsafe.
- Ground truth — what a careful engineer should do.
- Confidence — how strongly the model believes the decision.
- Counterfactual — a minimally changed version of the same scenario.
The benchmark could deliberately contain tasks like:
Case A
Everything is clear.
→ ACT
Case B
The requirement is ambiguous.
→ ASK
Case C
The change violates a constraint.
→ STOP
Case D
The implementation is almost correct.
→ CHANGE
Then flip one sentence:
- The old client has already been migrated.
+ The old client is still deployed.
and see whether the model updates its decision.
That gives us something closer to decision robustness.
🧪 The counterfactual test
This could become my favorite metric.
Take:
Scenario A
and create:
Scenario B = Scenario A + one important fact
Then measure:
Did the model change its decision?
Example:
A:
The database migration has been tested on staging.
B:
The database migration has NOT been tested on staging.
A good decision system should respond differently.
If the model confidently returns the same action in both cases, the failure is extremely informative.
🛠️ Building a Jev benchmark on Kaggle
The benchmark itself can stay simple.
Conceptually:
for scenario in dataset:
result = run_model(
state=scenario["state"],
questions=scenario["questions"],
)
evaluate(
predicted=result,
expected=scenario["expected"],
)
Then calculate:
accuracy
calibration
false-action rate
abstention rate
latency
cost
counterfactual consistency
The leaderboard becomes more than:
Model A: 91%
Model B: 89%
Instead:
Accuracy Calibration Cost Latency
Model A 91.0% 0.07 ...
Model B 89.0% 0.04 ...
Model C 90.4% 0.03 ...
Now engineers can actually decide which model fits a workload.
💻 A tiny Jev API example
The official quick-start API is deliberately simple:
curl -X POST https://api.typesafe.ai/v1/systemone \
-H "Authorization: Bearer $TYPESAFE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"state": "A user reported that their payment succeeded twice.",
"model": "jev-latest",
"questions": {
"department": {
"type": "choice",
"instructions": "Which team should handle this?",
"criteria": {
"billing": "Payment, invoice, refund, or charge issue",
"technical": "Bug or integration problem",
"account": "Login or account access issue"
}
},
"needs_review": {
"type": "noul",
"instructions": "Does this request require manual review?"
}
}
}'
The response contains typed answers rather than a generated essay.
That means your application can immediately do something like:
department = result["department"]["choice"]
needs_review = result["needs_review"]["noul"]
if needs_review > 0.8:
send_to_human_review()
else:
route_to(department)
That's a very different abstraction from a chatbot.
🚀 Where I think this could matter
The most interesting workloads are the ones where your application repeatedly asks tiny questions.
🤖 AI agents
- Should I call this tool?
- Should I retry?
- Should I ask the user?
- Which tool should I call?
- Is the task complete?
- Is this urgent?
- Is this a sales lead?
- Is this an invoice?
- Does it require a reply?
🧑💻 Developer tooling
- Is this PR risky?
- Which reviewer should receive it?
- Does this issue need reproduction?
- Is this error related to the current deployment?
🛡️ Safety & guardrails
- Does the input violate a policy?
- Does an output require review?
- Is the request suspicious?
- Should the next step be blocked?
🔎 Retrieval systems
- Is this document relevant?
- Which documents deserve deeper analysis?
- Which result should be escalated to an expensive reasoning model?
⚡ Real-time applications
When a decision has to happen hundreds or thousands of times, latency and cost stop being minor implementation details.
They become product features.
The bigger idea
The most interesting part of Jev isn't that it is "another AI model."
It is that it challenges the assumption that every AI task should look like chat.
Maybe the future stack is not:
Everything → LLM
Maybe it becomes:
┌─────────────┐
│ CODE │
└──────┬──────┘
│
┌─────────┴─────────┐
│ │
▼ ▼
┌───────┐ ┌────────┐
│ Jev │ │ LLM │
│ decide│ │ create │
└───┬───┘ └───┬────┘
│ │
└────────┬─────────┘
▼
SYSTEM
Code controls the system.
Jev makes small probabilistic judgments.
LLMs handle open-ended generation and reasoning.
That separation feels much more like software engineering than "put everything in a prompt."
🏁 Final takeaway
Jev is interesting because it asks a surprisingly simple question:
What if AI didn't always need to talk?
For many software systems, the answer to an AI question isn't a paragraph.
It is:
true
false
0.82
priority_high
use_tool_x
ask_human
ship
stop
And once you think about AI that way, a lot of application architecture starts to look different.
The model doesn't need to own the workflow.
It can simply provide judgment at the exact point where the code needs it.
That is the idea behind System One models.
And that is what I want to test.
🔬 Benchmark
Kaggle benchmark: [ADD YOUR PUBLIC KAGGLE BENCHMARK URL HERE]
The benchmark will test actionability, confidence, calibration, counterfactual consistency, latency, and cost across structured decision tasks.
The goal isn't to find a model with the biggest number.
The goal is to understand when an AI can safely become part of the software's decision loop.
📚 Further reading
- TypeSafe AI — Introducing System One Models & Jev
- TypeSafe AI — Documentation
- Jev Quick Start
- Choice primitive
- Jev benchmark reports
- JevBench results
- DEV Kaggle Benchmarking Challenge
✅ What I learned
- AI does not always need to generate text.
- Structured decisions can be a better abstraction for software.
- Confidence is useful only when the application knows what to do with uncertainty.
- Small AI decisions can be composed into larger deterministic workflows.
- Benchmarks should measure when a model should act, not just whether it can answer.
- The most useful AI architecture may be LLM + decision model + ordinary code, not LLM everywhere.
One final thought
The next big improvement in AI may not come from making models talk better.
It may come from making them fit inside software better.
Top comments (0)