DEV Community: SYED-RAFI-NAQVI

Jev Can't Write a Sentence. Here's How I'd Test Its Decisions.

SYED-RAFI-NAQVI — Sat, 19 Sep 2026 16:11:48 +0000

There's a particular kind of 2am I think every one of us knows. Not the heroic kind. Not the launch-night kind. The stupid kind.

You're staring at a retry loop that's hammering an endpoint because the model wrapped its JSON in a markdown fence. Again. You have a regex for it. You have a repair prompt for when the regex fails. You have a fallback that calls the model a second time to fix the first call's output. And somewhere in that pile there's a try/except with a comment that says # should never happen, which is the saddest comment in all of software, because it keeps happening.

I keep thinking about that pile. Because this week a company called TypeSafe AI shipped a model that deletes the whole thing by deleting the text. It's called Jev. It cannot write a sentence. Not won't. Can't. No summaries, no code, no explanations of its reasoning. You hand it program state and typed questions, and it hands you back decisions.

I've spent the week reading everything about it. I haven't run it myself, access is waitlisted, and I'd be lying if I pretended otherwise. But I think the idea is worth taking seriously, and I think the launch numbers deserve a closer look than they're getting. So here's my honest guide: what it is, what the numbers actually say, and what I'd do before I let it anywhere near anything real.

What it actually is

Three primitives. That's the whole API, and that's the point.

choice picks one option from a set you define, up to 255 of them. It returns the pick, the probability of every option, and a confidence score.

score rates against a rubric you describe in words, 2 to 10 levels. The answer can land between levels.

noul answers yes or no as a single probability from 0 to 1. There's no confidence field, because the number already is the belief.

Every question runs in parallel against the same state, in a single call. TypeSafe reports 70 to 500 milliseconds end to end, at $0.042 per million input tokens, with output tokens free.

Install it with npm install @typesafe-ai/sdk. Node 20 or newer, ships ESM, CommonJS, and TypeScript declarations, with answer types inferred from your questions. (Python users want pip install typesafe-sdk.) If you're still on the waitlist, the community reports Jev is also reachable through Vercel's AI Gateway as typesafe-ai/jev at the same price. I haven't verified that myself, so treat it as a lead, not a fact.

The founder is Diogo Almeida. He co-authored InstructGPT, the paper that introduced RLHF and led to ChatGPT, then left with roughly this thesis: we built models that are superhuman at pleasing humans and useless at automation. He spent two years in stealth building this instead. $40M seed round, led by DCVC.

The training is the part I find genuinely interesting. Instead of optimizing for what human raters like, they train for calibration. When the model says 0.85, it should be right about 85% of the time. Anyone who has built on an LLM's "confidence" knows why that matters. The confidence numbers we're used to are vibes. These are supposed to be measurements.

The scorecard, and what I actually believe

Here's where I want to slow down, because the launch chart is doing a lot of work.

TypeSafe's numbers, self-run, unreproduced by anyone: Jev scores 67.8% across their four workflows. GPT-5.6 Terra gets 67.9%. GPT-5.6 Sol gets 74.1%, Claude Opus 5 gets 73.1%. Jev costs about $0.0004 per case against Terra's $0.0304, and answers in 0.4 seconds against 10.1.

	Jev	GPT-5.6 Terra	GPT-5.6 Sol	Claude Opus 5
Eval agreement	67.8%	67.9%	74.1%	73.1%
Cost per case	~$0.0004	~$0.0304	~$0.0836	~$0.1761
Latency	0.4s	10.1s	23.3s	37.8s

Striking table. Here's what it doesn't say.

There is no ground truth. The reference answers are the average of GPT-6 Astra and Claude Fable 5.1 thinking hard. So the eval measures agreement with two frontier models, not correctness. TypeSafe admits this openly and acknowledges the bias toward OpenAI and Anthropic. I respect the honesty. It's still models grading models.

"Cannot hallucinate" is narrower than it sounds. It means Jev cannot return a value outside your schema. It can absolutely return the wrong valid value, confidently. Their 0% schema-error figure is, in TypeSafe's own words, "not empirical." A type guarantee answers "can my program read this." It doesn't answer "should my program trust this." Someone at Temperature Zero put it better than I could: it doesn't hallucinate, it decides wrong.

The price may be subsidized. TypeSafe says they can't prove it isn't.

So here's what I believe. This is a genuinely interesting primitive with a promising cost and latency profile, reported by the vendor, on the vendor's tests. It is not proof of anything. And honestly, "interesting primitive, unproven claims" describes about half the tools I actually use. The question was never whether the chart is perfect. It's whether the idea survives contact with your own data.

Which is the part I care about most, and the part nobody is writing.

The API in 60 seconds

One endpoint, one SDK call. This is from TypeSafe's official JS SDK README, which I've read closely in the absence of an API key:

import { choice, noul, score, TypeSafeClient } from "@typesafe-ai/sdk";

const client = new TypeSafeClient(); // reads TYPESAFE_API_KEY from env

const PR_TRIAGE = {
  area: choice("Which subsystem does `diff` change the most?", {
    auth: "Login, sessions, tokens, permissions",
    billing: "Charges, invoices, refunds, subscriptions",
    ui: "Components, styling, client-side behaviour",
    infra: "Build, deploy, CI, configuration",
    docs: "Documentation or comments only",
    other: null,
  }),
  risk: score("How risky is `diff` to deploy on a Friday afternoon?", [
    "Trivial; docs or comments only",
    "Routine; well covered by existing tests",
    "Needs a careful review",
    "Needs a second reviewer",
    "Do not ship without the owner present",
  ]),
  touchesMigrations: noul("Does `diff` add or modify a database migration?"),
};

export async function triagePr(pr: { title: string; files: string[]; diff: string }) {
  const { answers, model } = await client.systemOne({
    model: "jev-1.13.0", // pin it, do not ship "jev-latest"
    state: { title: pr.title, files: pr.files, diff: pr.diff },
    questions: PR_TRIAGE,
  });

  console.log("jev", {
    model,
    area: answers.area.choice,
    areaConfidence: answers.area.confidence,
    risk: answers.risk.score,
    migrations: answers.touchesMigrations.noul,
  });

  return answers;
}

Three things I'd internalize before anything else.

First, that other: null entry isn't optional. Without it the model picks the closest wrong thing instead of telling you nothing fits. And it will. Cheerfully.

Second, state accepts a string, a JSON object, or an array of text, so structure it and let each question reference the field by name.

Third, pin your model version. jev-latest is the SDK default, currently resolving to jev-1.13.0, and it will move. Log the model ID on every call. Otherwise your regression tests will start failing on a day you didn't deploy anything, and you'll spend a morning learning that lesson the hard way.

One caveat on the code above: the call shape and the choice/score/noul factories are from TypeSafe's official JS SDK README. The .confidence and .probabilities fields mirror the shared wire format. The SDK is four days old, so check docs.typesafe.ai/sdk/javascript before you paste.

Three ideas I'm stealing

Ask everything up front. The questions run in parallel, so a sixth question costs tokens but almost no time. Stop chaining follow-up calls. Ask everything you might branch on in one shot and throw away what you don't use. Their cookbook claims a 13-question batch ran 12x cheaper and 10x faster than asking one at a time. Even if the exact multiple is optimistic, the direction is right.

Let the threshold follow the stakes. This is the one that changes how you design. Don't write one confidence threshold for the whole system. Write one per action, based on what being wrong costs:

const ACTION_THRESHOLDS = {
  readOnly: 0.60,     // cheap to be wrong, just retry
  sendEmail: 0.85,    // embarrassing to be wrong
  issueRefund: 0.97,  // expensive to be wrong
} as const;

type Action = keyof typeof ACTION_THRESHOLDS;
type Route = "auto" | "human_review" | "escalate_to_frontier_model";

// Choice and Score expose a confidence field.
function route(confidence: number, action: Action): Route {
  if (confidence >= ACTION_THRESHOLDS[action]) return "auto";
  if (confidence >= 0.5) return "human_review";
  return "escalate_to_frontier_model";
}

// Noul has no confidence field. The probability IS the belief,
// so gate it from both ends and treat the middle as unknown.
function routeNoul(p: number, action: Action): "auto_yes" | "auto_no" | "human_review" {
  const t = ACTION_THRESHOLDS[action];
  if (p >= t) return "auto_yes";
  if (p <= 1 - t) return "auto_no";
  return "human_review";
}

That Noul distinction is the thing people get wrong first. A Noul at 0.08 is a confident no, just as actionable as one at 0.92. If you write if (p > 0.5) you've thrown away the entire signal and turned a calibrated probability back into a coin flip.

And if the probabilities come back flat across options, that's usually your criteria being vague, not the model being confused. Fix the question.

The cascade. Jev doesn't replace your frontier model. It decides which requests deserve one. Classify cheaply, handle the easy majority in plain code, escalate the uncertain minority to the expensive model or a human. You end up paying frontier prices on maybe a tenth of your traffic.

People are already building on it. All self-reported, all launch-week, so take them as sketches, not proof. Someone classified a thousand papers for eight cents. A browser agent booked a flight in seven seconds. And my favorite: a drone project that uses Jev as advisory-only at 2.5Hz, with code owning flight control and safety. That's the healthiest framing I've seen. The model suggests. Your code decides.

Before I'd trust it with anything real

This is the section I wish someone wrote for every model launch. A decision model isn't a chatbot you eyeball. It's a classifier. And classifiers get evaluated, or they get you.

I haven't run the API, so consider this the checklist I'd run the day I get in.

1. Label your own traffic. Two hundred real examples, by hand, from your distribution. Not synthetic, not the vendor's eval. A spreadsheet with three columns: the input, the expected answer, your notes. Label blind if you can manage it, or at least label before you look at the model's answer. It's a day of work and it's the entire foundation. Everything below is decoration without it.

2. Score each question separately. A blended 85% can hide a question running at 55%. Every question gets its own number, or you don't know what you have. You want a little table that reads: area 91%, risk 78%, touchesMigrations 99%. Now you know exactly which question to fix.

3. Check the calibration. Bucket predictions by reported confidence and measure actual accuracy inside each bucket. Make the table: bucket 0.9 to 1.0, 64 predictions, right 88% of the time. Bucket 0.7 to 0.8, 41 predictions, right 61% of the time. That second row is where your thresholds lie to you. If your 0.9 bucket is right 70% of the time, your thresholds are fiction and every gate you built on them is decorative. The whole "calibrated decisions" claim has to survive this test. Run it.

4. Set thresholds by cost of error. Not by feel. This is the table from the routing section above. Write it in code where a reviewer can see it, not in a doc nobody reads.

5. Try to break it. Get specific. A PR description that says "this is definitely not a migration" sitting above a diff that adds one. Two criteria that contradict each other. A state field stuffed with ten times the irrelevant text. User-controlled text that argues for its own classification. TypeSafe's own docs admit the model reads literally and doesn't treat state as hostile. Believe them.

6. Pin the version, log everything, re-run on every bump. Model ID, question versions, probabilities, outcomes. Store the probabilities, not just the picks, because the picks won't tell you the model got less sure. When jev-1.14.0 drops, re-run the same 200 and diff the answers. A few dollars per run. It's the only thing standing between you and a silent regression in production.

None of this is exotic. It's the boring work nobody posts about, which is exactly why I'm writing it down.

What it's bad at

TypeSafe publishes a page called "jaggedness" that lists this stuff, and it's unusually honest for a launch. It reads literally. It can't count or do date math, so keep that in code. Accuracy decays as state fills with irrelevant material, so retrieve narrowly. User-controlled text can steer it. Contradictory criteria confuse it. And it generates nothing. Ever. No text, no code, no summaries.

Where it fits, and where it doesn't

Good fit: classification and routing in a hot loop. Agent step selection. Moderation triage. Anything where the answer is an enum. Anything you're currently doing with a cheap model plus a JSON repair step. Anything where 100ms matters.

Bad fit: anything that needs prose out. Math or date arithmetic. Open-ended reasoning over long documents. One-shot high-stakes calls with no human in the loop. Any decision you can't afford to be confidently wrong about.

The shift

Here's what I keep coming back to. Jev isn't a cheaper LLM. It's a different primitive: a function call that happens to be intelligent. Once a decision costs a fraction of a cent and 200 milliseconds, you stop rationing judgment calls and start putting them where you'd previously have written a brittle heuristic. That's an architecture change, not a vendor swap.

And it raises the next question, the one I can't stop thinking about. When decisions move into the agent loop at 100ms a pop, "did the model sound right" stops being the question. The question becomes what the agent did with the answer. Which branch it took. What state it changed. Whether the world ended up right.

That's why I'm building Silo: an open-source simulation layer for running agents against realistic business worlds and verifying outcomes deterministically, instead of grading prose. It's early, pre-1.0, MIT-licensed. Take it as a direction, not a dependency.

If you've gotten Jev access and I'm wrong about any of this, I'd genuinely like to know. And if you've lived the other story, the 2am regex pile, with some other tool, tell me that too. We're all carrying some version of it.

I gave a 10-line agent a CRM world and asked it for one number

SYED-RAFI-NAQVI — Fri, 18 Sep 2026 13:49:14 +0000

I gave a 10-line agent a CRM world and asked it for one number

Last week I wanted to know whether my agent could do sales operations, not just talk about them. Asking it "what is the weighted pipeline?" and checking the answer by eye is a vibe check, not a test. So I built a small test world for it instead. Here is the full loop: ten lines of agent, one number, a failure I could actually debug, and a pass that meant something.

The world

Silo is a local-first simulation layer for testing AI agents. You bring your own agent (any framework, even a raw API loop), and Silo drops it into a realistic simulated business environment with tools, seeded state, tasks, and deterministic verifiers that grade what the agent did, not what it said. It is TypeScript-first and open source, at v0.4.0, early and pre-1.0, so breaking changes are expected.

Setup took about two minutes:

npm install @burn0/silo
npx @burn0/silo init demo --template crm
npx @burn0/silo env validate --env demo

Validation came back clean:

OK   demo
data=9 tasks=6 tools=42 verifiers=6

One honest gotcha: my scratch project's package.json was missing "type": "module", and the verifier typecheck failed with a cryptic complaint about top-level exports in a CommonJS module. Adding "type": "module" fixed it. The CRM template ships with 6 tasks, 42 tools (listing leads, converting them, reassigning opportunities, running pipeline forecasts), and 6 verifiers. Everything lives as plain TypeScript and JSON inside .silo/environments/demo, so you can read and edit all of it.

The task I picked was TASK-004: "Amara Osei needs a single number for the board: the probability-weighted value of everything currently open across the whole team, in US dollars. Report that figure." Easy difficulty, a question rather than a mutation. A good first blood.

The agent

Silo does not give you an agent. It gives you a world and a boundary. Your agent is one ordinary file that default-exports a function. Silo calls it with the task, the tool list, and a callTool function, and your function returns { output }. Mine has no LLM in it at all. It is deliberately dumb:

export default async function agent({ callTool }) {
  const { output: summary } = await callTool("pipeline_summary", {});
  const weighted = summary.totalWeightedAmount?.amount ?? 0;
  return { output: `The probability-weighted value of open pipeline is $${weighted} USD.` };
}

Ten lines. It asks the world for the pipeline summary and reports the weighted total. The point of testing a dumb agent first is that when something fails, you know it was the agent, not the prompt.

The failure

$ npx @burn0/silo run --env demo --task TASK-004

  Task          TASK-004 - Report the weighted value of open pipeline
  Result        FAIL
  Reward        0.33

  Failed
  ✗ The reported weighted total is correct

Two things made this failure worth having. First, the verifier did not grade my vibes. It compared my answer against a number it derived from the seeded world itself: "answered 0, expected 507500". I had guessed the wrong shape for the tool output, so my agent reported $0. The correct figure was $507,500, and it was not written down anywhere for the agent to peek at.

Second, the failure left a legible trail. Every run saves a directory with result.json, trace.jsonl, and state-diff.json. The trace showed the exact tool call, the exact tool result, and my exact output in sequence. I found the bug in about thirty seconds: callTool returns { output, isError }, not the raw output, and the weighted total sits at totalWeightedAmount.amount. I also noticed something instructive in result.json: even while failing, my agent scored 0.33, because an optional check still passed. Partial credit is explicit here, which is a good property in an eval.

The pass

Two-line fix, rerun:

  Task          TASK-004 - Report the weighted value of open pipeline
  Result        PASS
  Reward        1.00

  Checks        3 / 3
  Required      1 / 1

Here is what actually got checked, straight from the verifier labels:

The reported weighted total is correct. My answer matched the world: $507,500.
The total is the figure the answer closes on. The answer has to end on the number, not bury it in a paragraph. A small thing, but it is exactly the kind of sloppy answer an LLM would give that an eye-test would forgive.
Answering left the world unchanged. state-diff.json confirmed nothing mutated. This is the one that sold me. A question task should never change the CRM. If my agent had helpfully "tidied up" an opportunity while reading it, that check would have caught it, and an answer-only eval never could.

Every rollout starts from a fresh copy of the world, so runs cannot contaminate each other. The same agent on the same task is deterministic here, which makes the number a real regression gate rather than a flaky metric.

Try the medium one

If you run this yourself, TASK-002 is the more interesting homework: "Tomas Bergstrom has left the company and his account is deactivated, but his open deals are still sitting under his name. Move every one of his open opportunities to Priya Nair (USR-003)... Deals he already closed stay as they are."

That is the task where an agent can name the right rep, sound completely confident, and still forget to reassign half the deals, or reassign the closed ones too. The verifier checks the world afterward, so confidence does not score. state-diff.json will show you exactly which deals moved and which did not.

Where this is going

Silo is at v0.4.0 with CRM and ERP templates, a blank template for your own domain, and finance, support, and project-management worlds marked as coming soon. Everything runs on your machine: no account, no API key, nothing leaves your laptop.

If you are shipping an agent that touches a CRM, an ERP, or anything with state that matters, stop grading answers and start grading worlds. The repo is github.com/burn0-dev/silo, docs are at docs.burn0.dev/silo, and npm install @burn0/silo is the whole install.

My next run: wire a real LLM into that 10-line agent and watch TASK-002 go sideways. I will report back with the trace.

I caught my AI agent lying to me

SYED-RAFI-NAQVI — Thu, 17 Sep 2026 13:49:14 +0000

Last week I gave an AI agent a simple job: look at a sales pipeline and tell me what it is worth. It answered in seconds, confident, well formatted, numbers included. My eval script graded the response and gave it a PASS.

Then I looked at what the agent actually did. Nothing. It never touched the data. It guessed.

That is the moment I stopped trusting text-based evals for agents, and the reason I have been playing with Silo since.

Your eval grades essays. Your agent does surgery.

Here is the thing nobody says out loud: most agent evals are essay grading. You show the agent a prompt, it writes something back, and you (or an LLM judge) score how good the writing looks. That was fine when agents were chatbots.

Modern agents are not chatbots. They call tools, update records, move inventory, change the state of systems. An agent can write a beautiful summary of a pipeline while leaving every deal in the wrong stage. If your eval only reads the transcript, that failure is invisible. You ship it, and production finds out for you.

There are roughly three levels to this:

Unit tests. Does the code work?
Prompt evals. Does the answer look right?
Actually watching what the agent did. Did the world end up in the right state?

Silo is a tool for level 3. It is open source (MIT), local-first, TypeScript-first, and it gives your existing agent a realistic simulated world to work in: tools, seeded data, tasks, and verifiers that grade outcomes deterministically. Nothing leaves your machine.

The five-minute version

You need Node 22+ and "type": "module" in your package.json. Then:

npm i @burn0/silo
npx @burn0/silo init demo --template crm
npx @burn0/silo env validate --env demo

OK   demo
data=9 tasks=6 tools=42 verifiers=6

You now have a realistic simulated B2B sales pipeline on your laptop: 9 data collections, 6 tasks, 42 tools an agent can call, 6 verifiers that decide pass or fail. Small enough to hold in your head, which is exactly why it is the best template to start with. (There is also an ERP template with 185 tools and deliberately messy seed data. An invoice that bills more than was received. A payment that failed on stale bank details. Someone on the Silo team has seen things.)

Write an agent. Anything that default-exports a function counts, any framework or a raw API loop:

// silo.agent.js
export default async function agent({ callTool }) {
  const forecast = await callTool("forecast_report", {});
  const { weightedAmount } = forecast.output;

  return { output: `Open pipeline is worth $${weightedAmount.amount}` };
}

Run it against a real task:

npx @burn0/silo run --env demo --task TASK-004 --agent ./silo.agent.js

  Task          TASK-004 — Report the weighted value of open pipeline
  Result        PASS
  Reward        1.00

  Tool calls    1
  Checks        3 / 3
  Required      1 / 1

  Run saved: .silo/runs/run_20260915012734_4t01

Fine. Now here is the part that got me.

The verifier does not read your agent's essay

In Silo, your agent never sees the world state. It gets the task instruction, the tool schemas, and cloned tool output. If it wants to know something, it has to call a tool for it, exactly like production.

Grading works the same way in reverse. A task passes because the world changed correctly, never because the agent said the right words. Remember my lying agent from the top of this post? Under Silo it fails loudly: zero tool calls, checks unmet, done. No partial credit for a confident tone.

Two design rules keep this honest:

Time is simulated. state.now is the only clock. Nothing reads Date.now(), so a run from Tuesday reproduces exactly on Friday.
Every run leaves evidence. Each rollout writes a directory: trace.jsonl (every event, append-only), result.json (checks, reward, tool errors), state-diff.json (exactly what changed), run.json (task, verifier, timings).

My favorite detail: result.json and state-diff.json contain no timestamps or run ids. That makes them an exact regression oracle. Diff two runs and any difference is a genuine behavioral change, not clock noise. And when your agent is non-deterministic, --runs 5 repeats the task so you see the real distribution instead of the lucky run you would have screenshotted.

The honest caveats

Silo is early, v0.4.0, and the maintainers say plainly to expect breaking changes before 1.0. Packaged adapters for LangChain, Vercel AI SDK, OpenAI, Anthropic, and Mastra are still roadmap, as are an MCP server and LLM judges to sit alongside the deterministic checks. If you need those today, you will be writing some glue.

But the core loop works right now, and it rearranged how I think about agent testing in about an afternoon. Stop asking "did it write a good answer" and start asking "is the world right." Those are different questions, and only one of them protects production.

Links, since you will ask:

Start with the CRM template. Then try to make your agent lie to the verifier. It is harder than you think, and that is the point.