TL;DR
Model: Jev, from TypeSafe AI. The first "System One" model, launched September 2026.
What it is architecturally: Not an autoregressive generator. You declare a constrained output space and it returns the probability mass over that space. There are no output tokens in the generative sense, which is why they're free.
Output surface: Three primitives. Noul (a probability), Score (a position on your rubric plus its distribution), Choice (one of up to 255 options plus its distribution). No text, no coordinates, no pixels.
The property that changes your design: Every question in a request is evaluated in parallel against one shared read of the state. Eight questions cost roughly what one costs.
Trained with: RLCD (Reinforcement Learning for Calibrated Decisions), optimising against outcomes rather than human preference. This is why calibration is even a coherent claim here.
What I built: Semantic Microscope. Labels every sentence of a document with five or six calibrated probabilities and renders the whole thing as a heatmap.
My numbers (from Bengaluru): five documents, roughly 68,000 judgments, 31 cents, zero errors. p50 356ms round trip, of which only 98ms is the model and 247ms is the Pacific.
Measured calibration (n=59, hand-labelled): discrimination is excellent and monotonic, but the probabilities sit below the diagonal in every bucket. The cause turned out to be a disagreement about what the question meant, which is the most useful thing I learned.
Repo: github.com/abhishekmishragithub/semantic-microscope
A few days after Jev launched, a demo went round my timeline. Someone holds their hand up to a webcam, says "give me a ring", and a ring appears on their finger. Then sunglasses. Then a rainbow over their head. Credited to Jev. It did numbers.
I got access that week and started building, and about an hour in I realised something: Jev could not have drawn any of that. Not "it would be difficult". Structurally could not. There is no request you can make to Jev that produces a coordinate, an SVG path, or a pixel, because Jev's output space is something you declare in advance and it only ever returns a distribution over what you declared.
Whatever produced that ring was almost certainly MediaPipe finding the finger joint and a pre-made sprite being dropped on it. Jev's largest possible role was picking the string "ring" out of a list.
I'm not writing this to dunk on anyone (the demo is fun, and launch week is chaos for everybody). I'm writing it because the misunderstanding in that demo is the same one that will make your first Jev project disappointing. People are reaching for Jev as though it were a fast, cheap LLM. It's a different class of component, and once the architecture clicks you start seeing places for it that an LLM was never right for in the first place.
This post covers what Jev actually is under the hood, why its economics are shaped the way they are, the five patterns that work, where it fails, and real latency and calibration numbers I measured myself from India.
What kind of model is this, actually
Start here, because everything downstream follows from it.
An LLM is an autoregressive next-token predictor. You give it a context, it produces a probability distribution over the vocabulary, samples one token, appends it, and repeats. If you want a decision out of that process you have to convince it to write the decision down as tokens, then parse those tokens back into a decision. Structured output modes (JSON schema, grammar-constrained decoding) make the parsing more reliable, but the machinery underneath is unchanged: it's still generating a sequence, and you're still paying per token generated.
Jev inverts this. You declare the output space up front, as a Noul (binary), a Score (an ordered rubric of 2 to 10 levels), or a Choice (up to 255 options). The model does a forward pass over your state, and rather than sampling a sequence, it returns the probability mass over the space you declared.
TypeSafe hasn't published the architecture, so some of this is inference rather than documented fact. But the evidence is strong. openjev (166 stars) reproduces the primitive on a 4B open model by reading option logits directly, without generating anything, and gets something recognisably similar. That tells you the mechanism is a constrained read rather than a constrained generation.
Three consequences follow immediately, and they explain most of the strange things about Jev's pricing and API:
Output tokens are free because there aren't any. TypeSafe charges $0.042 per million input tokens and nothing for output. That's not a pricing promotion, it's a description of the architecture. Nothing is being generated, so there's nothing to meter.
Questions can be parallel because they don't share a sequence. Every question in a request is an independent read against the same forward pass over the same state. Question 4 cannot be influenced by question 2's answer, because there's no sequence for them to sit in. This is also why adding a question doesn't cause the context degradation you'd get from stuffing more into an LLM prompt.
Schema violations are structurally impossible, not merely unlikely. With an LLM in JSON mode you're constraining the sampler and hoping. With Jev the output space is your schema, so there is no state the model can be in that produces an invalid value. This is a real guarantee and it is also narrower than the marketing suggests, which I'll come back to.
Why calibration is a coherent claim here
The other half of the architecture story is the training objective. TypeSafe calls it RLCD, Reinforcement Learning for Calibrated Decisions: probabilities are optimised against observed outcomes rather than against human preference.
That distinction matters more than it sounds, and there's a well-known precedent. The GPT-4 technical report includes a pair of calibration plots showing the pre-trained base model was well calibrated on multiple-choice questions, and the post-RLHF model was substantially less calibrated. Training on human preference teaches a model that confident-sounding answers are rewarded, which is precisely the pressure that destroys calibration.
So when a chat model tells you it's "85% confident", that number is a linguistic artifact and means approximately nothing. When Jev returns 0.85, the training objective was at least pointed at making that number track reality.
"At least pointed at" is doing real work in that sentence. Calibration is a population-level property: if Jev says 0.8 across a thousand cases, roughly eight hundred should be true. It says nothing about whether this particular 0.8 is correct. And TypeSafe publishes no reliability diagram and no ECE, so this remains a claim about their training objective rather than a measured guarantee. I show you how to check it yourself later in this post, and I show you what I found.
The API, concretely
One endpoint. You send a state (a string or JSON object describing the situation) and a set of questions.
import httpx, os
r = httpx.post(
"https://api.typesafe.ai/v1/systemone",
headers={"Authorization": f"Bearer {os.environ['TYPESAFE_API_KEY']}"},
json={
"model": "jev-1.13.0",
"state": {
"sentence": "Either party may terminate upon thirty days written notice.",
"document_kind": "commercial contract",
},
"questions": {
"obligation": {
"type": "noul",
"instructions": "This sentence creates a binding obligation on one of the parties.",
},
"risk": {
"type": "score",
"instructions": "How legally load-bearing and unusual is this sentence?",
"criteria": [
"Pure boilerplate",
"Standard market terms",
"Materially binding",
"Unusual or one-sided",
],
},
"clause_type": {
"type": "choice",
"instructions": "Which category does this sentence belong to?",
"criteria": {
"termination": "Ending, suspending or renewing the agreement",
"payment": "Pricing, invoicing, fees or timing",
"liability": "Indemnity, warranty, limitation of liability",
"other": "None of the above",
},
},
},
},
)
Note the flat "type" form. I tried the nested {"noul": {...}} version first and got a 422.
What comes back:
- Noul returns a single float, the probability the proposition is true. No confidence field, because the probability is the belief.
-
Score returns a continuous value that can land between levels (
2.14), the full per-level distribution, a confidence, and a legend mapping indices back to your level text. The continuous value is a probability-weighted mean over the rubric, which is why it isn't an integer. - Choice returns the winning option, the full distribution across options, and a confidence derived from how peaked that distribution is.
Pin the model version (
jev-1.13.0) rather than usingjev-latest, and log themodelfield that comes back in every response. If you're tuning thresholds you really do not want the model moving under you mid-project.
A few operational limits worth knowing before you design anything: the state plus all questions share roughly a 64k token budget, Choice caps at 255 options, Score takes 2 to 10 levels, and the published rate limits are 250k tokens/sec and 1,200 requests/min. TypeSafe notes those limits are moving while capacity lands.
The mental model: a sense, not a reasoner
Kahneman's Thinking, Fast and Slow splits cognition in two. System 2 is slow, deliberate, sequential, the thing you use to work through a proof. System 1 is fast, automatic, parallel, and it does not produce arguments. It produces impressions. You walk into a room and know something is off before you could explain why.
Every LLM you've used is System 2 cosplay. It produces a chain of words, and if you want a decision you have to ask it to write one down and then read the words back.
Jev is shaped like System 1, and the architecture section above is why: parallel, non-sequential, no trace, output is an impression rather than an argument. There's no reasoning trace because there's no reasoning. That's the design, not a missing feature.
The practical version:
Stop asking what Jev can replace. Start asking where your system currently has no instincts at all.
Almost every LLM pipeline I've seen is a brain with no reflexes. It thinks carefully about everything, including things that should have been felt instantly. Those gaps are where Jev goes.
The demos are (mostly) misattributed
The ring isn't the only one. Once I started looking, a pattern showed up across the 180+ community projects that appeared in the first week. Almost all of them are real and clever. Almost all of them are credited to the wrong component.
"Jev drew captions and effects on my webcam." There's a Japanese project, jev-telop-live, that overlays variety-show captions and manga effects on a live camera feed. Its README is the most honest thing I've read about this model: it states outright that Jev does not write text, it answers typed questions with probabilities, and everything shown or played is picked from a closed set. The effects are hardcoded. Jev only selects which preset fires. That is exactly right, and exactly what the ring demo didn't say.
"Jev plays Doom / Mario / StarCraft / Pac-Man." Several of these exist and they do work. But Jev isn't seeing the screen. Something else serialises game state into text and Jev returns a Choice from a fixed action set each tick. Legitimate and interesting. The framing just makes people picture something that isn't happening.
"Jev booked a flight in 7.1 seconds for $0.0039." browser-use/jev-ultrafast, 641 stars, genuinely impressive. But browser-use did the browsing. Jev picked which element to click. The record belongs to the pair.
Published latency numbers from code that was never run. I found a Jev voice-agent repo quoting figures where the author notes, in the repo itself, that it had never been executed against the real API. Launch week produces a lot of this. Check whether numbers came from a real call before quoting them, including mine.
And the one that cuts the other way: openjev reproducing the primitive off a 4B model's logits, which I mentioned above. Someone rebuilt the shape of this in a weekend on open weights. That's genuinely useful information when you're deciding how much to build on the hosted version: the interface is copyable, and what TypeSafe is actually selling is the calibration training plus the hosting.
None of this means Jev is overhyped. It means the thing it's actually good at is less flashy and more useful than the demos suggest.
Four properties, and what each one buys
1. The output is a calibrated float, not a token stream. This is a category change, not a convenience. You get a number you can threshold, rank by, interpolate, animate, or send to a DAC. And you get gradations: not "risky" but 0.73. You can sort four hundred clauses by how risky they are. You cannot sort by the word "risky".
2. Questions are parallel and output is free. TypeSafe's cookbook reports batching 13 questions into one call was about 12× cheaper and 10× faster than asking them serially, with identical answers. My own runs match: going from 3 to 6 questions barely moved latency. So ask everything up front, including things you probably won't use. The LLM instinct of asking the cheap question first and escalating is exactly backwards here.
3. Under 100ms of model time. Fast enough to sit inside a loop. A render frame, a game tick, a conversation turn. Classification stops being a batch job and becomes a primitive you call like a function.
4. Effectively free. Five documents including two full novels, the GDPR and an IETF specification: roughly 68,000 judgments for 31 cents.
Jev vs an LLM vs a fine-tuned classifier
This is the comparison most posts dodge, because a fine-tuned BERT has been faster and cheaper than Jev at inference for years. The honest answer is that the difference isn't inference cost at all. It's what it costs to define a question.
A fine-tuned classifier needs labelled training data, hundreds to thousands of examples per class. You train, deploy, and have a model that answers exactly one question. Want a seventh dimension? Collect labels, retrain, redeploy. Reword a criterion? Retrain. Every question is a small project.
With Jev a question is a sentence of prose, defined at call time. I reworded one of my criteria three times in an afternoon and re-ran the whole document after each change, for a few cents total.
| Fine-tuned classifier | Jev | LLM | |
|---|---|---|---|
| Cost to add a question | Days | Seconds | Seconds |
| Needs training data | Yes | No | No |
| Inference latency | ~5-20ms | ~100ms server | 0.5-5s |
| Calibrated by design | No | Claimed | No (RLHF degrades it) |
| Can generate text | No | No | Yes |
| Cost at 10M/day | Lowest | Moderate | Prohibitive |
| Runs offline | Yes | No | Sometimes |
Use a fine-tune when the taxonomy is frozen, volume is enormous, you need single-digit milliseconds, or you must run on-prem. And note the natural progression: if a Jev question of yours stabilises and volume explodes, distilling it into a small classifier is a perfectly good endgame. You'll already have the labelled dataset, because Jev generated it.
Use Jev when questions change often, you have many of them, you have no labelled data, or you want probabilities rather than labels.
Use an LLM when you need words.
The one-line rule
If you are writing a prompt that ends with "respond only with one of: A, B, C", that is a Jev call, and you should stop writing that prompt.
Every JSON-mode enum. Every LLM-as-judge returning a 1 to 5. Every "is this safe, yes or no" guardrail. Every router picking a model. These are decisions wearing a text costume. You're paying generation prices and generation latency for something that was never text, then writing parsing code to undo it.
Five patterns that work
The gate. One Noul in front of an expensive or irreversible action. 100ms and a hundredth of a cent to avoid a bad outcome is trivially worth it, and unlike an LLM guardrail it cannot talk itself into a wrong answer through a long chain of reasoning.
The router. One Choice before an expensive model. The classic cascade, except the router is now three orders of magnitude cheaper than the thing it routes to.
The fan-out. Twenty Nouls in one call covering every safety, intent and policy flag you care about. Because the marginal question is free, you can afford to check for things that almost never happen. Most under-used pattern right now.
The reflex. Jev inside a real-time loop, running concurrently with something slow. In a voice agent that's the gap between STT finishing and TTS starting: your LLM drafts a reply while Jev simultaneously decides whether the user is frustrated, whether this is in scope, and whether a human should take over. The fast path lands before the slow path does.
The dense labeller. Nobody is building this one.
The pattern nobody is using: label everything
Put the four properties together. Many questions, free. Per-item cost near zero. Fast. Calibrated floats instead of labels.
That means you can afford to ask every question about every item in a corpus, and then treat the resulting probabilities not as decisions but as a field you can look at.
I built Semantic Microscope to test this. It splits a document into sentences and asks Jev five or six questions about every single one, then renders the whole document as parallel vertical strips, one row per sentence, coloured by probability. You don't read the output. You look at it. A forty-page contract becomes a picture and the three clauses that matter glow. Point it at a novel with a different question set and the three-act structure shows up as bands.
The core:
DIMENSIONS = {
"obligation": ("noul", "This sentence creates a binding obligation."),
"ambiguous": ("noul", "A reasonable professional could read this two incompatible ways."),
"financial": ("noul", "This sentence has a direct financial consequence."),
"liability": ("noul", "This limits, excludes, caps or assigns liability."),
}
async def label(sentence, prev, nxt):
state = {
"previous": prev,
"sentence": sentence,
"next": nxt,
"document_kind": "commercial contract",
}
return await jev.ask(state, build_questions(DIMENSIONS)) # one call, all dimensions
Two things here are easy to get wrong and both cost badly:
One request per item, all questions inside it. If you're looping over questions you've made the run 6× slower, 6× more likely to hit the rate limit, and thrown away the exact architectural property the approach depends on.
Send the item, not the document. One sentence plus one of context each side. Stuffing the whole document into state degrades accuracy as irrelevant material crowds it out, and burns input tokens (the ones you do pay for) for nothing. Filtering context to what the question needs is your job, not the model's.
Every existing Jev project I could find treats the output as a decision that triggers an action. Browser agents, trading bots, game players, routers, guardrails. Almost nobody treats the probability distribution as an artifact in its own right. That's open ground, and it's the thing this architecture makes possible that genuinely wasn't possible last month.
Two more worked examples
The interrupt gate
I run a daemon that speaks my coding agent's lifecycle events aloud. The failure mode of that whole genre is spam. One Noul fixes it:
questions = {
"worth_interrupting": {
"type": "noul",
"instructions": "A developer in deep focus would want to be told this immediately.",
},
"urgency": {
"type": "score",
"instructions": "How urgently does this need attention?",
"criteria": ["Ignore", "Mention later", "Say now", "Stop everything"],
},
}
Speak only above the bar, and pick voice and speed from the score. Turned a noisy toy into something I leave running.
Routing before the expensive model
questions = {
"skill": {
"type": "choice",
"instructions": "Which specialist should handle this request?",
"criteria": {
"code": "Writing or debugging software",
"research": "Needs current external information",
"chat": "Conversational, no tools needed",
"refuse": "Out of scope or unsafe",
},
},
"needs_frontier_model": {
"type": "noul",
"instructions": "This request requires the most capable model rather than a fast one.",
},
}
Route on choice, and use confidence to decide when to fall back to the expensive path rather than trusting the route. A flat distribution across four options is the model telling you it doesn't know, and that's actionable information you don't get from an LLM router.
Where it fails
- No arithmetic, no counting, no date ordering. Dates are text to it. Ask judgments, never computations.
- It reads literally, and that's mostly a feature. Negations and scoping are taken at face value. On RFC 9110 it read "non-compliant" as a MUST question and separated MUST (median 0.75) from SHOULD (0.33) almost cleanly. The cost is that you own the question wording completely: there's no prose in the response to reveal that it interpreted you differently.
- State is not treated as hostile. If user input goes into state, prompt injection is your threat model, not the model's.
- Context rot is real. Accuracy degrades as state fills with irrelevant material. The parallelism protects questions from each other, not from a bloated state.
- Served from US West only. Geography is a first-class latency cost if you're not in North America.
- "Cannot hallucinate" is narrower than it sounds. The guarantee is structural: it cannot return a value outside your declared space. It can absolutely return the wrong value from inside that space, with high confidence. TypeSafe's CEO said as much on Hacker News. Schema safety is not correctness.
- The headline benchmarks are vendor self-reported. "20 to 200× faster, 40 to 400× cheaper" and the accuracy figures are TypeSafe's own and unreproduced. The one independent hands-on test I've seen (Every) confirmed speed and cost, found accuracy "good but not perfect", and explicitly did not audit calibration.
My numbers, measured from Bengaluru
Vendor latency figures assume you're sitting near the inference pool. I'm 13,000km from it. Five documents, cold cache, 15 requests/s with 12 in flight, nothing rounded away:
| run | preset | sentences (live) | wall | p50 | p95 | p99 | errors | cost |
|---|---|---|---|---|---|---|---|---|
| sample-contract (synthetic MSA) | contract | 484 (482) | 33.2s | 385ms | 506ms | 617ms | 0 | $0.0141 |
| gdpr-contract (GDPR, EUR-Lex) | contract | 1,977 (1,630) | 108.9s | 357ms | 454ms | 578ms | 0 | $0.0481 |
| pride-and-prejudice (Gutenberg #1342) | prose | 4,533 (4,433) | 295.9s | 347ms | 439ms | 619ms | 0 | $0.1017 |
| rfc9110 (HTTP Semantics) | rfc | 3,566 (3,001) | 200.6s | 356ms | 446ms | 570ms | 0 | $0.0826 |
| frankenstein (Gutenberg #84) | prose | 3,087 (2,970) | 198.8s | 410ms | 515ms | 636ms | 0 | $0.0669 |
That's roughly 68,000 judgments across five documents for 31 cents, including two full novels, the GDPR, and an IETF specification. Zero errors and zero 429s across all of it.
Doing the same with a frontier model would have been a budget conversation rather than a rounding error.
Where the time actually goes
TypeSafe returns an x-envoy-upstream-service-time header, so you can separate inference from network. On the RFC 9110 run:
| component | p50 | p95 | p99 |
|---|---|---|---|
| server (from the envoy header) | 98ms | 162ms | 209ms |
| network and TLS (client minus server) | 247ms | 322ms | 367ms |
Roughly two thirds of every request is the Pacific, not the model. Jev itself answers in under 100ms. Two things follow:
Reuse your connections. A cold TLS handshake across that distance costs more than the inference. One persistent client for the whole run is not an optimisation, it's the difference between three minutes and most of an hour. I found this the annoying way.
Geography decides which patterns work for you. Dense labelling and gating are comfortable at 356ms. A reflex inside a live conversation turn, fighting for an 800ms end-to-end budget, is marginal from India and fine from Oregon. Measure before you design.
One more detail worth reading off that table: every run sat pinned at my own 15 rps limiter for its entire duration, so wall time is just sentences ÷ 15. The bottleneck was my rate limiting, not Jev. The documented ceiling is 1,200 requests/min and I was deliberately running under it.
And the cache matters more than I expected. Re-running the GDPR document with a warm cache took 2.9 seconds for 1,977 sentences and made zero requests. That turns "iterate on the viewer while re-running the pipeline" from a cost decision into a non-decision, which changed how I worked on this more than any other single thing.
Does the model actually track the question? A free check first
Before hand-labelling anything, there's a cheap sanity check worth running, and I haven't seen anyone else do it.
One of my six dimensions was deliberately lexical:
"normative_keyword": {
"type": "noul",
"instructions": "This sentence contains an RFC 2119 keyword: MUST, MUST NOT, SHALL, "
"SHOULD, SHOULD NOT, REQUIRED, RECOMMENDED, MAY, OPTIONAL.",
}
That question has objective ground truth. A regex answers it perfectly. So I ran the regex over all 3,001 sentences and compared.
Recall 1.00. Precision 0.79.
Jev found every single one of the 413 keyword sentences, and flagged 107 more that contain no uppercase keyword.
Look at what those 107 are, though. They're overwhelmingly sentences stating a requirement in lowercase prose: "a server must", "clients should". Which means Jev is doing exactly what a careful human reader does before being told the rules, and getting it wrong for exactly the reason the IETF had to publish RFC 8174, a whole RFC clarifying that only the uppercase forms carry normative force, because everyone kept conflating them.
Two things I'd take from that. First, a Noul does genuinely track its question rather than returning vibes. Second, "reads literally" is a real property and it cuts both ways: here it over-fires because the question said "contains a keyword" and Jev reasoned about meaning instead of characters.
If you're building on Jev, find the one question in your set that has objective ground truth and check it this way. It costs nothing and it's the fastest signal you'll get.
Measured calibration: the part nobody publishes
TypeSafe says Jev's probabilities are calibrated. They publish no reliability diagram and no ECE. So I measured it.
Method
The dimension:
"requirement": {
"type": "noul",
"instructions": "Ignoring this sentence would make an implementation non-compliant.",
}
The labelling rule, written before I saw any sentences and not revised during the run:
Work down the list, stop at the first match.
- contains MUST, MUST NOT, SHALL, SHALL NOT, REQUIRED → yes
- contains SHOULD, SHOULD NOT, RECOMMENDED → no
- contains MAY, OPTIONAL → no
- states an absolute requirement in plain prose, no keyword → yes
- only describes, defines or explains → no
- points at another section instead of stating a rule → no
SHOULD is a "no" because RFC 2119 defines it as permitting valid reasons to deviate. An implementation that ignores a SHOULD with good cause is still compliant, and the question asks about compliance.
Then: a stratified sample across the five probability buckets, with Jev's prediction hidden while labelling, in two sittings of 30. That last detail is the one people skip, and skipping it invalidates the whole exercise. If you see the number first, you're measuring your own anchoring.
Result (n=59, single annotator)
| bucket | n | predicted | observed |
|---|---|---|---|
| 0.0–0.2 | 12 | 0.15 | 0.00 |
| 0.2–0.4 | 12 | 0.31 | 0.00 |
| 0.4–0.6 | 12 | 0.46 | 0.08 |
| 0.6–0.8 | 11 | 0.71 | 0.45 |
| 0.8–1.0 | 12 | 0.82 | 0.75 |
Brier score: 0.165
Three findings, in order of how much they'd change your design.
1. Discrimination is excellent. Ranking is trustworthy.
The observed column rises monotonically: 0.00, 0.00, 0.08, 0.45, 0.75. No inversions anywhere. If you sort by the score, the sort is meaningful. For most real uses (triage, review queues, "show me the 30 that matter") this is the property you actually depend on, and it holds.
2. The probabilities sit below the diagonal in every bucket.
All five, same direction. Under a null of no bias that's about a 1-in-32 coincidence, and two buckets are individually significant: 0/12 against a predicted 0.31 has roughly a 1.2% binomial probability, and 1/12 against 0.46 about 0.8%.
Concretely, 0.71 meant about 0.45 here. But be precise about what's established: with 11 to 12 items per bucket the intervals are wide, and the two end buckets are individually consistent with calibration. It's the middle that isn't. The sign test across all five plus the two significant mid-buckets is the evidence, not any single row.
At n=29 after my first sitting I could see the shape but couldn't call it. The second sitting is what made it a finding rather than a hunch. If you run this yourself, do the second sitting.
This has an identifiable cause rather than being diffuse noise. I dig into it two sections down: the mid-range is almost entirely SHOULD sentences, which Jev scores at a median of 0.33 while my labelling rule scored them as hard no's.
3. The triage number, which is the one I'd actually put in a pitch deck.
Nothing below 0.4 was a real requirement. Zero out of 24 labelled items, across two buckets that together hold 2,059 of 3,001 sentences, or 68.6% of RFC 9110.
So the honest operational claim is: this discards roughly two-thirds of the specification, with no false negatives observed at this sample size, and you read the rest. That's a much better claim than "AI reads your spec for you", and unlike that one, it has a number behind it.
The rule this gives you
For this question on this document: threshold at 0.6. Treat below 0.4 as safely discardable. Trust the ordering, discount the absolute values.
What I am not claiming
One dimension, one document, one annotator, n=59, eleven to twelve items per bucket. The question's wording does real work, as the next section shows. I am not saying Jev is miscalibrated in general. I'm saying that on this task I measured a gap, that you should measure your own, and that the tooling to do it is in the repo:
uv run microscope validate <run-id> --dimension requirement
uv run microscope validate <run-id> --dimension requirement --add 30
Do it before you put a probability in front of anything consequential. And publish what you find either way. Right now almost nobody has, which is the only reason this section is interesting.
What the overconfidence actually was
The calibration gap above has a mechanism, and finding it was the most useful thing in this whole exercise.
RFC 2119 gives you three tiers of obligation. So I grouped every sentence by which uppercase keyword it contains and looked at what Jev gave them under the requirement question:
| keyword | n | min | p25 | median | p75 | max | share ≥ 0.5 |
|---|---|---|---|---|---|---|---|
| MUST / MUST NOT | 201 | 0.23 | 0.67 | 0.75 | 0.79 | 0.90 | 96% |
| SHOULD / SHOULD NOT | 119 | 0.13 | 0.25 | 0.33 | 0.42 | 0.65 | 8% |
| MAY | 104 | 0.12 | 0.21 | 0.27 | 0.36 | 0.74 | 6% |
Only four sentences contain both a MUST and a SHOULD, so the separation isn't an artefact of mixed sentences.
Jev is reading the question correctly. It asked itself whether ignoring this makes you non-compliant, concluded that's a MUST question, and separated the tiers almost cleanly: 96% of MUST sentences land above 0.5 against 8% of SHOULD. Threshold at 0.5 and you have, in effect, a MUST detector that also catches requirements stated in plain prose without any keyword at all.
But look at where SHOULD sits. Median 0.33, not 0.05. Jev treats SHOULD as a weak yes rather than a no. My labelling rule, written from RFC 2119, treated it as a hard no.
That single disagreement explains most of the calibration gap. The 0.2 to 0.4 bucket, where I measured a predicted 0.31 against an observed 0.00, is the SHOULD population almost exactly. The model isn't badly calibrated there so much as it's holding a slightly different theory of the question than I am. A lawyer would side with me. Plenty of engineers would side with Jev, because in practice ignoring a SHOULD does often mean your implementation is wrong.
I'd call that overconfidence, because under the question as I defined it those probabilities are too high. But it's overconfidence with a cause you can see and correct for, not noise.
The general lesson, which is the thing I'd actually hand someone building on this:
Your question wording isn't prompt engineering. It's the schema.
With an LLM, a disagreement about what you meant surfaces in the prose. You read the answer and think, that's not what I asked. Jev returns a float. There is nowhere for the disagreement to appear, so it emerges downstream as a calibration gap and you blame the model.
So before you trust a Noul: write down what a "yes" means precisely enough that a stranger could label from it, then group your results by some feature you already understand and check the distributions. If a class of items you expected near 0.1 is sitting at 0.33, that isn't noise. That's the model telling you it read the question differently, and it's the cheapest debugging signal you'll get.
Worth noticing that both of my useful findings came from the same move: group the results by something you already have ground truth for, and look at the distribution. The regex check told me a Noul tracks its question. The keyword grouping told me which question it thought it was answering. Neither needed a single hand-label.
Where I think this lands
The obvious commercial fit is anywhere the work is "check N criteria against M documents", because that cost used to scale as N×M and now it basically doesn't. Compliance review, contract triage, policy audit, and increasingly, auditing AI agent transcripts against a policy, continuously, at volume.
With one discipline attached: triage, not verdict. The honest pitch is "this takes a reviewer from four hundred clauses to the thirty they must actually read". It is not "this decides compliance". Given calibration is an unverified claim, anyone selling the second version is selling something they can't back.
The less obvious fit is the dense labelling section. Treating probabilities as a field rather than a trigger. Visualisation, instruments, art. Nobody's there yet, and I think that's where the interesting work is.
Jev's real contribution isn't that it's cheap. It's that it makes asking cheap, and when asking is free you ask about everything, and you find out that documents and conversations and codebases have shapes you were never able to see.
Go build something and tell me about it :)
Sources & References
- TypeSafe AI docs - primitives, limits, and their own guidance on confidence thresholds. The confidence page is more careful than the marketing and worth reading first.
- Semantic Microscope - the tool from this post. Python pipeline, vanilla JS viewer, no framework, bring your own key.
- openjev - reproduces the primitive by reading option logits off a 4B open model. The best available evidence for how the mechanism actually works.
- jev-telop-live - the most honest README in the ecosystem about what Jev does and doesn't do.
- browser-use/jev-ultrafast - the flight booking demo. Good example of Jev as one component rather than the whole system.
- jevmeter - a live probability meter over a debate video, 1,191 calls for about 5 cents. Prior art if you're doing anything with probability as a visual.
- RFC 9110 - HTTP Semantics. The document everything in the measurement sections was run against.
- RFC 2119 - defines MUST, SHOULD, MAY and the rest. The reason SHOULD counts as "no" in my labelling rule.
- RFC 8174 - the clarification that only uppercase keywords are normative. Jev makes the same mistake this RFC exists to correct.
- GPT-4 Technical Report - see the calibration plots showing RLHF degrading the base model's calibration. Useful background for why RLCD is a different bet.
Top comments (0)