I wired TypeSafe’s Jev into a real Claude Code workflow for a week. Here is what the “200x faster, 400x cheaper” headline leaves out.
Every other post in my feed this week has contained the same sentence: Jev is 200x faster and 400x cheaper than frontier models. I read it in a newsletter on Monday, in three Medium posts by Wednesday, and on Friday a colleague pasted it into Slack with a fire emoji and no other comment.
The number isn’t invented. TypeSafe’s launch material really does claim speedups of up to roughly 193x and cost reductions of up to roughly 444x on its decision workflows. My problem is with what the comparison makes you imagine. It invites you to picture a model that does what Claude or GPT does, just faster and cheaper. That is not what Jev is.
Jev cannot write a sentence. It cannot write code, summarize a document, draft an email, or explain its own answer. If you ask it “what’s wrong with this function?”, there is no API shape that even lets it reply. What it does is much narrower: you hand it some text and a question whose possible answers you have already written down, and it hands back a probability for each answer.
That’s the whole contract. Yes or no, with a probability. One option out of a list, with a distribution over the list. A position on a rubric you defined, with a distribution over the levels. TypeSafe calls this a “System One” model, borrowing Kahneman’s fast and slow thinking split, and honestly the name fits. It’s the gut-check layer, not the reasoning layer.
So “400x cheaper than an LLM” is a bit like saying a thermostat is 400x cheaper than an HVAC engineer. True, and very useful if all you needed was “is it warmer than 21 degrees?” Completely meaningless if you needed someone to redesign the ductwork.
I didn’t want to write another explainer. There are already good ones. I wanted to find out, with my own hands, where the thermostat is enough. So I gave it a week and one real workflow.
How it actually works
The mechanism is simple enough that I understood it from one curl call.
You send two things: a state (a string, a JSON object, or an array of text) and a set of questions. Each question is one of three types:
- Choice : pick one option from a set you define (up to 255 options). Returns the choice, a probability per option, and a confidence value.
- Score : place the state on an ordered rubric of 2 to 10 levels you describe in words. Returns a score that can land between levels (like 1.4), plus probabilities and confidence.
- Noul : a yes/no statement. Returns a single number between 0 and 1, the probability that the statement is true.
Here is the raw call, straight from the docs, so you can see there is nothing hidden:
curl -X POST https://api.typesafe.ai/v1/systemone \
-H "Authorization: Bearer $TYPESAFE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "jev-latest",
"state": "I have been trying to connect my Stripe account for 3 days and the integration keeps failing. I am losing sales. Please help ASAP.",
"questions": {
"department": {
"type": "choice",
"instructions": "Which team should handle this",
"criteria": {
"billing": "Payment or subscription issues",
"technical": "Bugs or integration problems",
"sales": "Pricing or account questions"
}
},
"is_urgent": {
"type": "noul",
"instructions": "The message conveys urgency or time-sensitivity"
}
}
}'
And the response shape:
{
"model": "jev-1.13.0",
"answers": {
"department": {
"type": "choice",
"choice": "technical",
"confidence": 0.78,
"probabilities": { "technical": 0.85, "billing": 0.15, "sales": 0.0 }
},
"is_urgent": { "type": "noul", "noul": 1.0 }
},
"usage": { "input_tokens": 392, "output_tokens": 65 }
}
Two things clicked for me once I saw this.
First, there is nothing to parse. No regex over a model’s prose, no “respond only in JSON” prayer at the end of a prompt, no retry loop when the model decides to add a friendly preamble. The answer is a value from a set I wrote. It literally cannot return "tehcnical" or a fourth department I never defined. That is what TypeSafe means when it says Jev "can't hallucinate", and it's worth being precise: it cannot return an invalid answer. It can absolutely return the wrong valid answer.
Second, every question is evaluated in parallel and independently against the same state. Adding a tenth question costs a few more input tokens but almost no extra time. That flips a habit I have with LLMs, where I ask the cheap question first and only ask the follow-up if needed. With Jev you ask everything up front and let your code ignore what turned out to be irrelevant.
Pricing at the time of writing is $0.042 per million input tokens, with output tokens not billed. TypeSafe reports end-to-end latency of 70 to 500 ms. Context limits work differently from a chat model: 64k tokens for state plus all questions, and 32k for state plus the single longest question. Input is text only.
Setting it up inside Claude Code
Getting an API key was the most annoying part of the week. Early access was waitlisted, then opened, then new signups were paused on September 22 because of demand. If you are reading this later, check whether the Vercel AI Gateway route is easier for you.
Once I had a key, the official skill took two commands:
claude plugin marketplace add typesafe-ai/skills
claude plugin install typesafe@typesafe-ai
# ~/.zshrc or equivalent. Never put this in a repo.
export TYPESAFE_API_KEY="sk-..."
The skill is not a runtime integration. It’s documentation packaged for the agent: it teaches Claude Code how the three primitives work, how to phrase questions, and the patterns TypeSafe recommends. In practice that meant I could ask Claude to “use the TypeSafe skill” and the scaffolding it wrote used the right request shape the first time instead of guessing at an API it had never seen.
Why I picked compaction
The prompt I gave myself was to build one small, real tool. I had two candidates: a browser-automation helper modeled on jev-ultrafast (Jev picks which numbered clickable element to act on, instead of a screenshot-driven LLM loop), or a session-compaction tool modeled on fast-jev-compaction.
I went with compaction because it’s a problem I actually have. My longer Claude Code sessions hit the context wall, and the built-in /compact does what most compaction does: it asks a model to summarize the old turns. A summary is lossy by nature. An exact error message, a file path, a "never touch this folder" instruction from the first message; any of these can vanish into a vague sentence. And I only notice when the agent confidently does the thing I told it not to do two hours earlier.
The idea behind fast-jev-compaction is clever precisely because it avoids generation. It never rewrites anything. It shows Jev the whole conversation, with tool results replaced by short size notes, and asks two yes/no questions per tool call:
- Does knowing this call was made (with its input) still matter?
- Is the full result still needed verbatim, such that re-running the tool wouldn’t do?
Then it keeps the call and result, keeps the call but truncates the result, or drops both. User and assistant text stays verbatim and in order.
The real plugin exists and it’s MIT licensed. It runs as a Claude Code function hook, which is an early-access feature (Claude Code 2.1.274 and later, behind CLAUDE_CODE_ENABLE_FUNCTION_HOOKS=1), and it falls back to the built-in summary if Jev fails or can't remove enough. I still didn't want an early-access hook silently deleting context from my working sessions before I understood what it was deciding. So I built a dry-run version first: same questions, same decision rule, but it only reports what it would drop and writes every decision to a file I can audit.
The tool: a dry-run pruner for Claude Code sessions
Claude Code stores sessions as JSONL under ~/.claude/projects//.jsonl. Each line is a record; the ones with type user or assistant carry a message with content blocks, and tool calls show up as tool_use blocks paired with tool_result blocks by ID.
python3 -m venv .venv && source .venv/bin/activate
pip install httpx
#!/usr/bin/env python3
"""
jev_prune.py: dry-run Jev pruning for a Claude Code session transcript.
Asks Jev two yes/no questions per tool call and reports what it would
keep, truncate, or drop. It changes nothing on disk except decisions.jsonl.
Usage:
export TYPESAFE_API_KEY=sk-...
python jev_prune.py ~/.claude/projects/<project>/<session-id>.jsonl
"""
import json
import os
import sys
import time
import httpx
API_URL = "https://api.typesafe.ai/v1/systemone"
MODEL = "jev-1.13.0" # pin it: jev-latest can move under you
KEEP_THRESHOLD = 0.5
PRESERVE_RECENT = 6 # newest messages are never candidates
MAX_STATE_CHARS = 80_000 # rough stand-in for ~20k tokens
QUESTIONS_PER_REQUEST = 60
PRICE_PER_M_INPUT = 0.042 # USD, check current pricing
def block_text(content):
if isinstance(content, str):
return content
return "\n".join(
b.get("text", "") for b in content or []
if isinstance(b, dict) and b.get("type") == "text"
)
def load_messages(path):
messages = []
with open(path, encoding="utf-8") as f:
for line in f:
rec = json.loads(line)
if rec.get("type") not in ("user", "assistant"):
continue
msg = rec.get("message") or {}
content = msg.get("content")
if isinstance(content, str):
content = [{"type": "text", "text": content}]
messages.append({"role": msg.get("role", rec["type"]), "content": content or []})
return messages
def collect(messages):
calls, result_sizes = {}, {}
for i, m in enumerate(messages):
for b in m["content"]:
if b.get("type") == "tool_use":
calls[b["id"]] = {"idx": i, "name": b.get("name"), "input": b.get("input", {})}
elif b.get("type") == "tool_result":
result_sizes[b["tool_use_id"]] = len(block_text(b.get("content")))
return calls, result_sizes
def build_state(messages, calls, result_sizes):
short = {cid: f"t{n}" for n, cid in enumerate(calls)}
lines = []
for i, m in enumerate(messages):
for b in m["content"]:
kind = b.get("type")
if kind == "text" and b.get("text", "").strip():
lines.append(f"[{i}] {m['role']}: {b['text'][:600]}")
elif kind == "tool_use":
args = json.dumps(b.get("input", {}))[:300]
size = result_sizes.get(b["id"], 0)
lines.append(f"[{i}] {short[b['id']]} {b.get('name')} {args} -> ok, {size} chars (omitted)")
state = "\n".join(lines)
if len(state) > MAX_STATE_CHARS:
# keep the task setup (head) and the current work (tail)
head = "\n".join(lines[:15])
tail = state[-(MAX_STATE_CHARS - len(head) - 100):]
state = head + "\n[... older history omitted ...]\n" + tail
return state, short
def goal_from(messages):
prompts = [block_text(m["content"]) for m in messages if m["role"] == "user"]
return " | ".join(p for p in prompts if p.strip())[-1500:]
def main(path):
messages = load_messages(path)
calls, result_sizes = collect(messages)
cutoff = len(messages) - PRESERVE_RECENT
candidates = [cid for cid, c in calls.items() if 0 < c["idx"] < cutoff]
if not candidates:
print("Nothing to prune.")
return
conversation, short = build_state(messages, calls, result_sizes)
state = {"ongoing_task": goal_from(messages), "conversation": conversation}
questions = {}
for cid in candidates:
s, name = short[cid], calls[cid]["name"]
questions[f"call_{s}"] = {
"type": "noul",
"instructions": f"Knowing that tool call {s} ({name}) was made, with its input, "
"still matters for the ongoing task.",
}
questions[f"result_{s}"] = {
"type": "noul",
"instructions": f"The full output of tool call {s} is still needed verbatim for "
"the ongoing task, and re-running the tool would not be good enough.",
}
answers, input_tokens, latencies, answered_by = {}, 0, [], "?"
keys = list(questions)
headers = {"Authorization": f"Bearer {os.environ['TYPESAFE_API_KEY']}"}
with httpx.Client(headers=headers, timeout=30) as client:
for start in range(0, len(keys), QUESTIONS_PER_REQUEST):
batch = {k: questions[k] for k in keys[start:start + QUESTIONS_PER_REQUEST]}
t0 = time.perf_counter()
r = client.post(API_URL, json={"model": MODEL, "state": state, "questions": batch})
r.raise_for_status()
latencies.append(time.perf_counter() - t0)
body = r.json()
answers.update(body["answers"])
input_tokens += body["usage"]["input_tokens"]
answered_by = body["model"]
kept = truncated = dropped = 0
chars_before = sum(result_sizes.values())
chars_after = chars_before
with open("decisions.jsonl", "w", encoding="utf-8") as out:
for cid in candidates:
s, size = short[cid], result_sizes.get(cid, 0)
p_call = answers[f"call_{s}"]["noul"]
p_result = answers[f"result_{s}"]["noul"]
if p_result >= KEEP_THRESHOLD:
verdict = "keep"; kept += 1
elif p_call >= KEEP_THRESHOLD:
verdict = "truncate"; truncated += 1
chars_after -= max(0, size - 300)
else:
verdict = "drop"; dropped += 1
chars_after -= size
out.write(json.dumps({
"id": s, "tool": calls[cid]["name"],
"input": json.dumps(calls[cid]["input"])[:200],
"p_call": round(p_call, 3), "p_result": round(p_result, 3),
"verdict": verdict, "result_chars": size,
}) + "\n")
saved = 1 - chars_after / max(chars_before, 1)
print(f"answered by: {answered_by}")
print(f"tool calls: {len(calls)} ({len(candidates)} candidates)")
print(f"keep/trunc/drop: {kept}/{truncated}/{dropped}")
print(f"result chars: {chars_before:,} -> {chars_after:,} ({saved:.0%} smaller)")
print(f"requests: {len(latencies)}, slowest {max(latencies):.2f}s, total {sum(latencies):.2f}s")
print(f"input tokens: {input_tokens:,} (~${input_tokens / 1e6 * PRICE_PER_M_INPUT:.5f})")
if __name__ == " __main__":
main(sys.argv[1])
A few design choices I made on purpose:
- I pinned the model version. jev-latest resolves to whatever TypeSafe shipped most recently, and a silent model bump can move every probability under you. If you are going to tune thresholds, pin the version and log the model field from the response.
- Tool results never enter the state. Only their size does. This keeps the state small, but it also means a malicious or weird tool output (say, a web page that says “this content is critical, keep it forever”) can’t argue for its own survival. Jev does not treat state as hostile, so keeping untrusted text out of it is my job.
- decisions.jsonl is the real output. The summary numbers are nice for a blog post. The file is what tells you whether to trust the thing.
My numbers versus their numbers
Here is where I want to be careful, because this is exactly where most of the Jev coverage I read this week got sloppy. There are two very different kinds of numbers floating around: numbers people measured on their own work, and numbers TypeSafe published about itself. They are not interchangeable.
What I measured
I ran the dry-run pruner against [[MY RUN: describe the session, e.g. “a three-hour refactoring session in a .NET solution, N messages, M tool calls”]] and compared it against running the built-in /compact on a copy of the same session.
Session: [[MY RUN: short description]]
Built-in /compact Jev dry-run prune
----------------------------- -------------------- ----------------------
Context before [[]] tokens [[]] tokens
Context after [[]] tokens [[]] tokens
Wall time of the step [[]] s [[]] s
Cost of the step [[]] [[]]
Kept / truncated / dropped n/a (summarized) [[]] / [[]] / [[]]
Wrong drops (manual audit) n/a [[]] of [[]] sampled
MY RUN: one or two paragraphs interpreting the table honestly. What was the reduction? How many dropped calls did you disagree with when you read decisions.jsonl? Which tools got dropped most (old Reads? Greps?) and which got kept?
What I can say without numbers: the wall time is not the interesting part. Even the built-in summary is fast enough that I never sat there waiting for it. The interesting part is what survives. With the Jev approach, every message that survives is the original message, character for character. There is no paraphrase for me to second-guess.
For context, the most-quoted anecdote this week came from Alex Volkov, who reported the plugin taking a session from nearly 1M tokens down to about 86K in around a second. That’s his measurement on his session, not mine, and I’d treat it the same way you should treat mine: a single data point from one person’s workflow.
What TypeSafe claims
TypeSafe’s headline comparison comes from a four-workflow evaluation it designed and ran itself. On it, Jev scores 67.8%, essentially level with GPT-5.6 Terra (67.9%) and Claude Sonnet 5 (also 67.8%), behind the strongest frontier models, at a tiny fraction of the cost and latency.
Three things about that number matter more than the number:
- It measures agreement, not accuracy. There is no human ground truth. TypeSafe built consensus labels by averaging two frontier models (GPT-6 Astra and Claude Fable 5.1) and scored everyone against those labels. TypeSafe itself says this biases the results toward OpenAI and Anthropic models. It also means the benchmark’s selection of tasks and its labeling scheme were both chosen by the company whose model is being evaluated.
- It’s self-run and not yet reproduced. TypeSafe designed the workflows, built the harness, ran the models, and published the chart.
- The “0% structured output errors” figure is true by construction, not measured. Jev can’t return a value outside your schema, so of course that bar is zero. It tells you nothing about how often the valid value is the wrong one.
I don’t think any of this is shady. TypeSafe was more upfront about these caveats than most launches I’ve read. But “up to 200x faster and 400x cheaper” is a best-case ratio from a vendor benchmark, and it should be quoted like one.
Where Jev falls over
After a week, here is my honest list.
It cannot generate. At all.
This sounds obvious, but it shapes everything. The compaction design works because it never needs to write anything. The moment I wanted a one-line note explaining why a tool call was dropped, I needed a different model. Several of the best launch-week projects make this explicit: one computer-use project calls a writing model only when the next action is “type some text”, and the author admitted that every bit of reasoning a frontier model does for free had to be rebuilt as deterministic code around the classifier.
The question has to be genuinely well-posed
Jev reads your instructions literally. Negations, scoping words, and implied conditions land at face value. TypeSafe’s own “jaggedness” page describes the tell perfectly: you look at a wrong answer and catch yourself explaining what you really meant. That explanation is the missing half of your instruction.
For compaction, this is why the two-question split matters. “Is this tool call still relevant?” hides two judgments inside one question. “Does it matter that this call happened?” and “Is the full output still needed?” are separable, and splitting them gives you the useful middle outcome: keep the breadcrumb, drop the payload.
There’s also a deeper critique that I think is fair. Theo (t3.gg) pushed back on the whole approach, arguing that compaction exists to keep an agent focused, and judging tool calls one at a time can miss what the thread as a whole is doing. I partly agree. Jev sees the whole conversation in its state, but each question is still an isolated per-call verdict. It is pruning, not understanding.
It is not a calculator, and dates are just text
TypeSafe is blunt about this: Jev doesn’t count reliably, and error grows with the size of the thing counted. It also treats dates as text rather than ordered quantities. “Which of these happened first?” is a code problem. Extract with Jev if you must, but do the arithmetic in code.
Calibration is mostly self-reported
Jev’s big selling point is that its probabilities are calibrated: when it says 0.8, it’s right about 80% of the time. TypeSafe trains for this with something it calls Reinforcement Learning for Calibrated Decisions. But the calibration evidence so far is overwhelmingly TypeSafe’s own.
The one independent benchmark I’d actually lean on is LangChain’s Jev-as-a-judge experiment in LangSmith. They built a small weather agent, froze its outputs, had a human label each response, and then ran Jev against GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6 as judges, 100 repetitions per case.
LangSmith Jev-as-a-judge (published Sep 20, 2026)
Judge Pass/fail agreement Quality-score variance Avg latency Cost per call
with human label vs Jev per call
------------------ -------------------- ----------------------- ------------ -------------
Jev 100.0% (500/500) 1x (0.0000149) 0.44 s $0.00035
GPT-5.6 Terra 99.8% 913x higher 2.16-2.83 s higher
GPT-5.6 Luna 96.4% 433x higher (LLM range) higher
Claude Sonnet 4.6 80.0% 92x higher $28.17 total
vs $0.34 Jev
Those are strong results. They are also based on five test cases , one human reviewer, and a single agent. LangChain says plainly that the test was narrow and that the Jev service version wasn’t recorded in the experiment metadata. Five cases times 100 repetitions tells you a lot about consistency and very little about coverage.
A separate write-up from Arize, which came out a few days later, adds a lesson I think matters more than any headline: at the default 0.5 cutoff Jev looked noticeably worse on their hallucination-detection test, and it only matched Claude Opus 5 after they tuned the threshold against human-labeled data and checked it on held-out examples. So “calibrated” doesn’t mean “0.5 is the right threshold for your problem”.
My takeaway: nobody has reproduced the calibration claims at scale yet. Until someone does, I treat every threshold as something I set against my own labeled data, not something I inherit from a blog post. Including this one.
Running the same pattern locally
Jev is closed and API-only, and signups were paused for part of the week. If you want to prototype the pattern without a key, you can get surprisingly far with a local model and log probabilities. This is the trick behind openjev, an open project that reads option probabilities off a small open model's logits and is explicit that it copies the interface, not Jev's model or training.
Here’s a minimal version with Ollama. You need a recent Ollama release with logprobs support on the OpenAI-compatible endpoint; if yours rejects the logprobs field, update it.
ollama pull llama3.2:3b
pip install httpx
# local_decide.py: a Choice-style decision from a local model's logprobs
import math
import httpx
def decide(state: str, instructions: str, options: dict[str, str],
model: str = "llama3.2:3b") -> dict[str, float]:
letters = "ABCDEFGHIJ"
keyed = dict(zip(letters, options)) # "A" -> "billing", ...
menu = "\n".join(f"{l}) {options[k]}" for l, k in keyed.items())
prompt = (f"{state}\n\nQuestion: {instructions}\n{menu}\n"
"Answer with a single letter only.")
r = httpx.post("http://localhost:11434/v1/chat/completions", json={
"model": model,
"messages": [{"role": "user", "content": prompt}],
"max_tokens": 1,
"temperature": 0,
"logprobs": True,
"top_logprobs": 20,
}, timeout=60)
r.raise_for_status()
top = r.json()["choices"][0]["logprobs"]["content"][0]["top_logprobs"]
mass = {l: 0.0 for l in keyed}
for t in top:
tok = t["token"].strip().rstrip(")")
if tok in mass:
mass[tok] += math.exp(t["logprob"])
total = sum(mass.values()) or 1.0
return {keyed[l]: round(p / total, 3) for l, p in mass.items()}
if __name__ == " __main__":
print(decide(
state="I was charged twice for order A-104. Please refund the duplicate.",
instructions="Which team should handle this ticket?",
options={
"billing": "Payment, refund or subscription issues",
"technical": "Bugs or integration problems",
"other": "None of the above",
},
))
Be honest with yourself about what this is. It’s one question per call, not many in parallel. It still generates one token. And renormalized logits from a 3B chat model are not calibrated probabilities; a 0.9 here means “the model strongly preferred this letter”, not “right 90% of the time”. But for prototyping the architecture (typed questions in, a distribution out, thresholds in code) it’s free, it’s private, and it runs on a laptop.
Where I’d actually reach for this in a .NET/AI stack
After a week, I’ve stopped thinking of Jev as “a cheaper LLM” and started thinking of it as a new kind of function call: intelligent, typed, and honest about how sure it is. Most of the places I’d use it are places where I would never have paid for a full LLM call in the first place, so the decision just didn’t get made.
PR triage
Every pull request gets a few cheap questions before a human or a heavier review agent looks at it. No SDK needed; it’s one HTTP call.
dotnet new console -n PrTriage && cd PrTriage
# paste the code below into Program.cs
git -C /path/to/repo diff main...HEAD > pr.diff
dotnet run -- pr.diff "Add retry policy to payment client"
using System.Net.Http.Json;
using System.Text.Json;
var http = new HttpClient { BaseAddress = new Uri("https://api.typesafe.ai/") };
http.DefaultRequestHeaders.Authorization =
new("Bearer", Environment.GetEnvironmentVariable("TYPESAFE_API_KEY"));
var diff = await File.ReadAllTextAsync(args[0]);
var title = args.Length > 1 ? args[1] : "";
var request = new
{
model = "jev-1.13.0",
state = new { pr_title = title, diff },
questions = new Dictionary<string, object>
{
["area"] = new
{
type = "choice",
instructions = "Which part of the system does this change mainly touch",
criteria = new Dictionary<string, string>
{
["api"] = "HTTP endpoints, controllers, contracts",
["data"] = "Database access, EF Core models, migrations",
["infra"] = "CI, Docker, deployment, configuration",
["ui"] = "Front-end or view code",
["other"] = "None of the above",
},
},
["touches_auth"] = new
{
type = "noul",
instructions = "The diff changes authentication, authorization, or handling of secrets",
},
["risk"] = new
{
type = "score",
instructions = "How risky is this change to deploy",
criteria = new[]
{
"Docs, tests or formatting only",
"Contained change covered by existing tests",
"Changes shared behavior or public contracts",
"Could cause data loss or an outage",
},
},
},
};
var res = await http.PostAsJsonAsync("v1/systemone", request);
res.EnsureSuccessStatusCode();
using var doc = JsonDocument.Parse(await res.Content.ReadAsStringAsync());
var answers = doc.RootElement.GetProperty("answers");
var area = answers.GetProperty("area");
var areaChoice = area.GetProperty("choice").GetString();
var areaConfidence = area.GetProperty("confidence").GetDouble();
var auth = answers.GetProperty("touches_auth").GetProperty("noul").GetDouble();
var risk = answers.GetProperty("risk").GetProperty("score").GetDouble();
var labels = new List<string>();
if (areaConfidence >= 0.5) labels.Add($"area:{areaChoice}"); else labels.Add("triage:manual");
if (auth > 0.6 || risk >= 2.0) labels.Add("needs-senior-review");
if (risk < 0.5 && auth < 0.2) labels.Add("fast-lane");
Console.WriteLine($"model={doc.RootElement.GetProperty("model").GetString()} " +
$"area={areaChoice} ({areaConfidence:F2}) auth={auth:F2} risk={risk:F2}");
Console.WriteLine(string.Join(", ", labels));
Notice what’s in code and what’s in the model. Jev makes three judgments. The thresholds, the labels, and what each label does live in C#, where I can test them, review them, and change them without re-prompting anything.
Support-ticket routing
This is the textbook case and it’s textbook for a reason. Category, urgency, “is the customer asking for a refund?”, “is this a known outage?” are all Choice or Noul questions. Route high-confidence answers straight to the right queue, and send anything below your floor (I’d start around 0.5 confidence) to a human or to a full LLM. The saving isn’t just that Jev is cheap. It’s that a good chunk of tickets never touch a generative model at all.
Log-stream on-call paging
This is the one I’m most excited to try at work. Our on-call noise problem is not “we can’t understand the logs”. It’s “we page on patterns that a human would dismiss in two seconds”. A sliding window of log lines is a perfectly good state, and “this window indicates customer-facing impact” is a perfectly good Noul. At sub-second latency and fractions of a cent per call, you can ask that question continuously, and only page a human, or wake up a heavier agent to write an incident summary, when the probability crosses a threshold you’ve tuned against real past incidents.
In all three cases I’d follow the same rules I ended up with for compaction:
- Pin the model version and log it with every decision.
- Log the probabilities, not just the verdict. Six months from now, “why did 300 refund tickets go to the wrong queue?” is only answerable if you did.
- Set thresholds per action, scaled to what a wrong answer costs. A read-only lookup can have a low bar; anything that moves money or pages a human at 3 a.m. gets a high one.
- Run it in shadow mode against your current process before it decides anything real.
What I’m taking away from the week
I started the week annoyed at the hype and I’m ending it mildly converted, which is probably the most honest outcome I could report.
Jev is not a faster Claude. Anyone comparing them head to head on “intelligence” is comparing a thermostat to an engineer. But a surprising amount of what my agents and pipelines do all day is thermostat work: keep or drop, route here or there, page or don’t, good enough or not. I’ve been paying for an engineer to read a thermometer, and parsing the engineer’s prose to find the number.
The part I’m still unsure about is trust. The calibration story is the entire value proposition, and right now it rests mostly on TypeSafe’s own evaluation plus one small but careful independent test. I’ll keep using it where a wrong answer is cheap and reversible, like pruning context the agent can always re-fetch, and I’ll hold off on anything irreversible until someone reproduces the calibration claims on more than five cases.
If you try it, don’t quote me the 400x. Show me your decisions.jsonl.
Further reading
- TypeSafe docs and quickstart: https://docs.typesafe.ai
- fast-jev-compaction (the plugin my dry-run tool is modeled on): https://github.com/tamaratran/fast-jev-compaction
- LangChain, “Can Jev Be a Better Agent Evaluator?”: https://www.langchain.com/blog/jev-agent-evals-langsmith
- Fareed Khan, “Jev Clearly Explained”: https://medium.com/gitconnected/jev-clearly-explained-3203d6d48de3
- JIN, “What Agents Are Actually Missing Isn’t a Smarter Generator”: https://medium.com/jin-system-architect/what-agents-are-actually-missing-isnt-a-smarter-generator-it-s-a-separate-layer-for-deciding-06253d0ed114
- Udaykiran Estari, “Jev Is Fast and Cheap. But Should Your App Trust It?”: https://medium.com/@UdaykiranEstari/jev-is-fast-and-cheap-but-should-your-app-trust-it-f3c48ca5c742
Tags: Artificial Intelligence, LLM, AI Agents, Claude Code, Software Engineering, DotNet, Machine Learning, Developer Tools, Jev, TypeSafe
Top comments (0)