Jev Can't Write a Sentence. Here's How I'd Test Its Decisions.

There's a particular kind of 2am I think every one of us knows. Not the heroic kind. Not the launch-night kind. The stupid kind.

You're staring at a retry loop that's hammering an endpoint because the model wrapped its JSON in a markdown fence. Again. You have a regex for it. You have a repair prompt for when the regex fails. You have a fallback that calls the model a second time to fix the first call's output. And somewhere in that pile there's a try/except with a comment that says # should never happen, which is the saddest comment in all of software, because it keeps happening.

I keep thinking about that pile. Because this week a company called TypeSafe AI shipped a model that deletes the whole thing by deleting the text. It's called Jev. It cannot write a sentence. Not won't. Can't. No summaries, no code, no explanations of its reasoning. You hand it program state and typed questions, and it hands you back decisions.

I've spent the week reading everything about it. I haven't run it myself, access is waitlisted, and I'd be lying if I pretended otherwise. But I think the idea is worth taking seriously, and I think the launch numbers deserve a closer look than they're getting. So here's my honest guide: what it is, what the numbers actually say, and what I'd do before I let it anywhere near anything real.

What it actually is

Three primitives. That's the whole API, and that's the point.

choice picks one option from a set you define, up to 255 of them. It returns the pick, the probability of every option, and a confidence score.

score rates against a rubric you describe in words, 2 to 10 levels. The answer can land between levels.

noul answers yes or no as a single probability from 0 to 1. There's no confidence field, because the number already is the belief.

Every question runs in parallel against the same state, in a single call. TypeSafe reports 70 to 500 milliseconds end to end, at $0.042 per million input tokens, with output tokens free.

Install it with npm install @typesafe-ai/sdk. Node 20 or newer, ships ESM, CommonJS, and TypeScript declarations, with answer types inferred from your questions. (Python users want pip install typesafe-sdk.) If you're still on the waitlist, the community reports Jev is also reachable through Vercel's AI Gateway as typesafe-ai/jev at the same price. I haven't verified that myself, so treat it as a lead, not a fact.

The founder is Diogo Almeida. He co-authored InstructGPT, the paper that introduced RLHF and led to ChatGPT, then left with roughly this thesis: we built models that are superhuman at pleasing humans and useless at automation. He spent two years in stealth building this instead. $40M seed round, led by DCVC.

The training is the part I find genuinely interesting. Instead of optimizing for what human raters like, they train for calibration. When the model says 0.85, it should be right about 85% of the time. Anyone who has built on an LLM's "confidence" knows why that matters. The confidence numbers we're used to are vibes. These are supposed to be measurements.

The scorecard, and what I actually believe

Here's where I want to slow down, because the launch chart is doing a lot of work.

TypeSafe's numbers, self-run, unreproduced by anyone: Jev scores 67.8% across their four workflows. GPT-5.6 Terra gets 67.9%. GPT-5.6 Sol gets 74.1%, Claude Opus 5 gets 73.1%. Jev costs about $0.0004 per case against Terra's $0.0304, and answers in 0.4 seconds against 10.1.

	Jev	GPT-5.6 Terra	GPT-5.6 Sol	Claude Opus 5
Eval agreement	67.8%	67.9%	74.1%	73.1%
Cost per case	~$0.0004	~$0.0304	~$0.0836	~$0.1761
Latency	0.4s	10.1s	23.3s	37.8s

Striking table. Here's what it doesn't say.

There is no ground truth. The reference answers are the average of GPT-6 Astra and Claude Fable 5.1 thinking hard. So the eval measures agreement with two frontier models, not correctness. TypeSafe admits this openly and acknowledges the bias toward OpenAI and Anthropic. I respect the honesty. It's still models grading models.

"Cannot hallucinate" is narrower than it sounds. It means Jev cannot return a value outside your schema. It can absolutely return the wrong valid value, confidently. Their 0% schema-error figure is, in TypeSafe's own words, "not empirical." A type guarantee answers "can my program read this." It doesn't answer "should my program trust this." Someone at Temperature Zero put it better than I could: it doesn't hallucinate, it decides wrong.

The price may be subsidized. TypeSafe says they can't prove it isn't.

So here's what I believe. This is a genuinely interesting primitive with a promising cost and latency profile, reported by the vendor, on the vendor's tests. It is not proof of anything. And honestly, "interesting primitive, unproven claims" describes about half the tools I actually use. The question was never whether the chart is perfect. It's whether the idea survives contact with your own data.

Which is the part I care about most, and the part nobody is writing.

The API in 60 seconds

One endpoint, one SDK call. This is from TypeSafe's official JS SDK README, which I've read closely in the absence of an API key:

import { choice, noul, score, TypeSafeClient } from "@typesafe-ai/sdk";

const client = new TypeSafeClient(); // reads TYPESAFE_API_KEY from env

const PR_TRIAGE = {
  area: choice("Which subsystem does `diff` change the most?", {
    auth: "Login, sessions, tokens, permissions",
    billing: "Charges, invoices, refunds, subscriptions",
    ui: "Components, styling, client-side behaviour",
    infra: "Build, deploy, CI, configuration",
    docs: "Documentation or comments only",
    other: null,
  }),
  risk: score("How risky is `diff` to deploy on a Friday afternoon?", [
    "Trivial; docs or comments only",
    "Routine; well covered by existing tests",
    "Needs a careful review",
    "Needs a second reviewer",
    "Do not ship without the owner present",
  ]),
  touchesMigrations: noul("Does `diff` add or modify a database migration?"),
};

export async function triagePr(pr: { title: string; files: string[]; diff: string }) {
  const { answers, model } = await client.systemOne({
    model: "jev-1.13.0", // pin it, do not ship "jev-latest"
    state: { title: pr.title, files: pr.files, diff: pr.diff },
    questions: PR_TRIAGE,
  });

  console.log("jev", {
    model,
    area: answers.area.choice,
    areaConfidence: answers.area.confidence,
    risk: answers.risk.score,
    migrations: answers.touchesMigrations.noul,
  });

  return answers;
}

Three things I'd internalize before anything else.

First, that other: null entry isn't optional. Without it the model picks the closest wrong thing instead of telling you nothing fits. And it will. Cheerfully.

Second, state accepts a string, a JSON object, or an array of text, so structure it and let each question reference the field by name.

Third, pin your model version. jev-latest is the SDK default, currently resolving to jev-1.13.0, and it will move. Log the model ID on every call. Otherwise your regression tests will start failing on a day you didn't deploy anything, and you'll spend a morning learning that lesson the hard way.

One caveat on the code above: the call shape and the choice/score/noul factories are from TypeSafe's official JS SDK README. The .confidence and .probabilities fields mirror the shared wire format. The SDK is four days old, so check docs.typesafe.ai/sdk/javascript before you paste.

Three ideas I'm stealing

Ask everything up front. The questions run in parallel, so a sixth question costs tokens but almost no time. Stop chaining follow-up calls. Ask everything you might branch on in one shot and throw away what you don't use. Their cookbook claims a 13-question batch ran 12x cheaper and 10x faster than asking one at a time. Even if the exact multiple is optimistic, the direction is right.

Let the threshold follow the stakes. This is the one that changes how you design. Don't write one confidence threshold for the whole system. Write one per action, based on what being wrong costs:

const ACTION_THRESHOLDS = {
  readOnly: 0.60,     // cheap to be wrong, just retry
  sendEmail: 0.85,    // embarrassing to be wrong
  issueRefund: 0.97,  // expensive to be wrong
} as const;

type Action = keyof typeof ACTION_THRESHOLDS;
type Route = "auto" | "human_review" | "escalate_to_frontier_model";

// Choice and Score expose a confidence field.
function route(confidence: number, action: Action): Route {
  if (confidence >= ACTION_THRESHOLDS[action]) return "auto";
  if (confidence >= 0.5) return "human_review";
  return "escalate_to_frontier_model";
}

// Noul has no confidence field. The probability IS the belief,
// so gate it from both ends and treat the middle as unknown.
function routeNoul(p: number, action: Action): "auto_yes" | "auto_no" | "human_review" {
  const t = ACTION_THRESHOLDS[action];
  if (p >= t) return "auto_yes";
  if (p <= 1 - t) return "auto_no";
  return "human_review";
}

That Noul distinction is the thing people get wrong first. A Noul at 0.08 is a confident no, just as actionable as one at 0.92. If you write if (p > 0.5) you've thrown away the entire signal and turned a calibrated probability back into a coin flip.

And if the probabilities come back flat across options, that's usually your criteria being vague, not the model being confused. Fix the question.

The cascade. Jev doesn't replace your frontier model. It decides which requests deserve one. Classify cheaply, handle the easy majority in plain code, escalate the uncertain minority to the expensive model or a human. You end up paying frontier prices on maybe a tenth of your traffic.

People are already building on it. All self-reported, all launch-week, so take them as sketches, not proof. Someone classified a thousand papers for eight cents. A browser agent booked a flight in seven seconds. And my favorite: a drone project that uses Jev as advisory-only at 2.5Hz, with code owning flight control and safety. That's the healthiest framing I've seen. The model suggests. Your code decides.

Before I'd trust it with anything real

This is the section I wish someone wrote for every model launch. A decision model isn't a chatbot you eyeball. It's a classifier. And classifiers get evaluated, or they get you.

I haven't run the API, so consider this the checklist I'd run the day I get in.

1. Label your own traffic. Two hundred real examples, by hand, from your distribution. Not synthetic, not the vendor's eval. A spreadsheet with three columns: the input, the expected answer, your notes. Label blind if you can manage it, or at least label before you look at the model's answer. It's a day of work and it's the entire foundation. Everything below is decoration without it.

2. Score each question separately. A blended 85% can hide a question running at 55%. Every question gets its own number, or you don't know what you have. You want a little table that reads: area 91%, risk 78%, touchesMigrations 99%. Now you know exactly which question to fix.

3. Check the calibration. Bucket predictions by reported confidence and measure actual accuracy inside each bucket. Make the table: bucket 0.9 to 1.0, 64 predictions, right 88% of the time. Bucket 0.7 to 0.8, 41 predictions, right 61% of the time. That second row is where your thresholds lie to you. If your 0.9 bucket is right 70% of the time, your thresholds are fiction and every gate you built on them is decorative. The whole "calibrated decisions" claim has to survive this test. Run it.

4. Set thresholds by cost of error. Not by feel. This is the table from the routing section above. Write it in code where a reviewer can see it, not in a doc nobody reads.

5. Try to break it. Get specific. A PR description that says "this is definitely not a migration" sitting above a diff that adds one. Two criteria that contradict each other. A state field stuffed with ten times the irrelevant text. User-controlled text that argues for its own classification. TypeSafe's own docs admit the model reads literally and doesn't treat state as hostile. Believe them.

6. Pin the version, log everything, re-run on every bump. Model ID, question versions, probabilities, outcomes. Store the probabilities, not just the picks, because the picks won't tell you the model got less sure. When jev-1.14.0 drops, re-run the same 200 and diff the answers. A few dollars per run. It's the only thing standing between you and a silent regression in production.

None of this is exotic. It's the boring work nobody posts about, which is exactly why I'm writing it down.

What it's bad at

TypeSafe publishes a page called "jaggedness" that lists this stuff, and it's unusually honest for a launch. It reads literally. It can't count or do date math, so keep that in code. Accuracy decays as state fills with irrelevant material, so retrieve narrowly. User-controlled text can steer it. Contradictory criteria confuse it. And it generates nothing. Ever. No text, no code, no summaries.

Where it fits, and where it doesn't

Good fit: classification and routing in a hot loop. Agent step selection. Moderation triage. Anything where the answer is an enum. Anything you're currently doing with a cheap model plus a JSON repair step. Anything where 100ms matters.

Bad fit: anything that needs prose out. Math or date arithmetic. Open-ended reasoning over long documents. One-shot high-stakes calls with no human in the loop. Any decision you can't afford to be confidently wrong about.

The shift

Here's what I keep coming back to. Jev isn't a cheaper LLM. It's a different primitive: a function call that happens to be intelligent. Once a decision costs a fraction of a cent and 200 milliseconds, you stop rationing judgment calls and start putting them where you'd previously have written a brittle heuristic. That's an architecture change, not a vendor swap.

And it raises the next question, the one I can't stop thinking about. When decisions move into the agent loop at 100ms a pop, "did the model sound right" stops being the question. The question becomes what the agent did with the answer. Which branch it took. What state it changed. Whether the world ended up right.

That's why I'm building Silo: an open-source simulation layer for running agents against realistic business worlds and verifying outcomes deterministically, instead of grading prose. It's early, pre-1.0, MIT-licensed. Take it as a direction, not a dependency.

If you've gotten Jev access and I'm wrong about any of this, I'd genuinely like to know. And if you've lived the other story, the 2am regex pile, with some other tool, tell me that too. We're all carrying some version of it.