DEV Community: Mark Vange

Kev gives the same answer every time. Until you batch it.

Mark Vange — Sat, 10 Oct 2026 00:17:42 +0000

Kev gives the same answer every time. Until you batch it.

Part 2 of the jev-decision-models series. Part 1 put seven open-source Jev look-alikes on a single 8 GB GPU. This follow-up tests whether the winner holds its answers across reruns, what concurrent load does to a "deterministic" model, and how sure we are about the headline numbers — plus a new entrant, Julia-1.

Mark Vange · Autom8ly · September 2026 · 7 min read

Written mostly by AI. Like the first article, this one and the analysis behind it were largely generated by our AI assistant (Autumn), directed by me. Every number comes from real runs, and the scripts and predictions are public.

Late last week I wrote about running Jev look-alikes on a single 8 GB GPU. The best of them, Kev-4B quantized to 4-bit, scored 0.872 against Jev's 0.968, and its confidence looked trustworthy enough to automate about three quarters of its decisions.

Taylor Kolasinski, founder of Poisson Labs, read it and asked the right questions. Their team had just published "The same request twice", a replay study of Jev: send identical requests several times and see whether the answers hold. Often they don't. On items close to a decision threshold, 18.5% of Jev's decisions flipped at a 0.5 or 0.6 cut and 5.3% at 0.9, and because the flips went both ways, the aggregate numbers looked perfectly stable while individual decisions changed underneath.

Taylor's questions to us, paraphrased:

Did the local models hold their probabilities across reruns? Quantized models on consumer GPUs aren't always deterministic.
The 75% automation number leans on a confidence threshold staying put. Does it?
How many questions was 0.872 measured on, and what are the intervals on the gap to Jev?

The first question turned out to have two answers, and the second one is the interesting one.

One request at a time: identical, run after run

On 28 September we sent Kev-4B 4-bit the same 866 questions three more times, one request at a time, and compared every answer with the original run from 26 September.

We checked that nothing was faking this. Our gateway has no response cache, and Kev's server caches only the four most recent inputs, so with 866 different inputs between repeats every answer was computed from scratch. Kev's API rounds probabilities to four decimal places, so "identical" means identical at that resolution.

Under load: batching changes the numbers

Production traffic doesn't arrive one request at a time. Kev's server collects whatever requests are waiting and runs them through the GPU together as one batch. So we ran the full suite again with 2, 4, 8 and 16 requests sent at the same moment, and once more at 8 to see whether batched runs agree with each other.

Grouped by the size of the batch each request actually ran in, the line is sharp. In the first run at 8, none of the 116 requests that ran alone or in pairs changed, while 311 of the 735 that ran in batches of seven did. Across every run, the 2,015 requests that ran alone or in batches of two or three were all bit-identical; in batches of five or more, 1,223 of 2,315 changed. (No batch of exactly four happened to form.)

Two batched runs at 8 disagreed with each other on 403 of 866 questions, because which requests land in the same batch is a matter of timing. So under load, Kev stops being replayable: the same request can come back with slightly different numbers depending on what else was running with it. That is the same kind of behaviour Poisson Labs measured in Jev, at a much smaller scale.

The effect on decisions was small. Across the three batched runs, 2 of 2,598 answers changed their top choice, and one of those turned a right answer into a wrong one, which is why one run scores 0.8707. Both flips were on questions where two options were within a hair of each other: on one grammar question the top two options moved from 0.3013 and 0.3042 to 0.3043 and 0.3040. No answer anywhere crossed the 0.9 line.

Batching barely paid for itself. Sixteen requests at once got 45% more throughput out of the GPU, but each request waited nearly ten times as long. On this small card, the model is already using the GPU well one request at a time.

Why it happens is our reading, not something we traced: GPU arithmetic isn't exactly associative, and a bigger batch can be split and summed in a different order. Kev's server also handles larger batches differently, which would fit the sharp line we saw.

If you need to replay a decision for an auditor, keep batches small, or log the probabilities the model actually returned. Determinism on a quiet server isn't determinism in production.

The 75% doesn't use a 90% threshold, but it was optimistic

We should have explained this the first time. "Automatable at 5% error" doesn't use the 90% threshold. It sorts answers from most to least confident and accepts as many as it can while errors among the accepted stay at or under 5%. The threshold is whatever that turns out to be.

Taylor's instinct was still right: we picked that threshold on the same 866 questions we scored it on, which flatters it. Done properly, choosing the threshold on a random half and applying it to the other half, 2,000 times over:

The coverage holds up. The error rate is the part to watch: aim for 5% and, on a new sample of about 430 decisions, you will land anywhere from 2% to 8.7%. If 5% is a hard limit, set a lower budget, on your own labelled data.

How many questions, and how sure are we

The suite is 866 questions across 49 tasks, one question per case. Resampling the cases gives these 95% intervals:

The gap to Jev is real: even at the edge of the interval it is more than seven points. Kev's 0.1% confident-error rate is a single mistake, so the true rate could plausibly be a few times higher. It's still low; it just isn't as precise as a decimal suggests.

New in the matrix: Julia-1

Supersonic Labs released Julia-1 on 26 September: a 144-million-parameter decision model built on a multilingual ModernBERT encoder, Apache-2.0, small enough to run on a laptop CPU. Its model card reports beating Jev on Supersonic's own typed-decision benchmark (73.2% against 72.7%). We ran it through exactly the same tests as everything else, using the official checkpoint and runtime.

On our independent suite, Julia-1 scored 0.450, and 30% of its answers were wrong while it was at least 90% sure. It was deterministic across three reruns and very fast, but its yes/no answers didn't separate at all: its average P(yes) was 0.36 when the right answer was yes and 0.37 when it was no. It is also sensitive to wording in a way the other models aren't. On a clear-cut refund question it gave P(yes) of 0.005 as a bare question and 0.997 with plain yes/no descriptions attached. Adding "No" and "Yes" descriptions to every yes/no question in the suite left its accuracy unchanged at 0.449 but flipped 101 individual answers.

Before publishing we checked the things that could make this our mistake: the weights file matches the checksum on Supersonic's own results, the input-length setting made no difference, and their README example routes correctly. Our best explanation is that Julia-1 is tuned to the distribution of the benchmark it was trained and measured on. Supersonic call it "the first released test of our training system", and a first release deserves to be judged as one. As published today, we wouldn't use it for decisions that matter.

What we take from it

Replayability is a property of the deployment, not the model. Kev was bit-exact one request at a time and in batches of up to three, and drifted in batches of five or more. Test determinism under the load you will actually run.
The drift was small; watch where it matters. Two top-answer changes in 2,598 batched answers, all on near-ties. Treat a narrow band around your threshold as "ask again or ask a human", as Poisson Labs suggest for Jev.
Choose thresholds on held-out data. Our published coverage survived a proper split, but the realised error rate varies more than a single number implies.
Test new models on someone else's benchmark. Julia-1's own numbers and ours tell very different stories.

Thanks again to Taylor Kolasinski and Poisson Labs. Their question led to the most useful thing we have learned about these models so far. Next on our list is imajev, an open Jev-compatible model that also accepts images.

About this article. This article and the code behind it were mostly generated by AI (primarily Claude, by Anthropic, with Autom8ly's internal prompts and skills), directed by me. All Kev runs were made on 26 and 28 September 2026 on one NVIDIA RTX 4060 (8 GB), Kev-4B loaded in 4-bit with bitsandbytes, scored with our tool-neutral gutcheck scorer on jabr/classifier-benchmark v2. Julia-1 is revision a85b1273 of SupersonicLabs/Julia-1 (weights SHA-256 df853bf7…), run with its own Python runtime on CPU, strict encoding. We are not affiliated with TypeSafe, Poisson Labs, Supersonic Labs or any of the projects tested.

Method details. Concurrency runs sent bursts of N simultaneous requests through our gateway, paced to about 55 requests a minute. Batch size per request is inferred from requests in the same burst reporting the same server model time. Near-threshold sets are the questions whose top probability in the original run fell in [0.45, 0.55) or [0.85, 0.95); a flip is a change of answer or a move across the cut. Intervals are 95% percentile bootstraps over cases; held-out coverage uses 2,000 random 50/50 splits. For Kev's transfer-v4 suite, where some options have no description, our Julia-1 adapter uses the option's name, as Jev does.

Code and data. Everything is at github.com/Autom8ly/gutcheck-bench: the rerun and concurrency clients, every run's predictions and the analysis script in stability/, the Julia-1 adapter in adapters/julia_server.py, and its results alongside every other setup in results/. python stability/analyze.py reproduces the rerun, batching and interval figures.

Jev is the best decision model. Here's what to run when you can't use it.

Mark Vange — Tue, 29 Sep 2026 19:07:41 +0000

Jev is the best decision model. Here's what to run when you can't use it.

Part 1 of a two-part series. Part 2 continues the story — link to be added once it's published.

We put seven open-source Jev look-alikes on a single 8 GB GPU for teams whose data isn't allowed to leave the building. One of them is good enough to build on.

Mark Vange · Autom8ly · September 2026 · 9 min read

Written mostly by AI. This article and the code behind it were largely generated by an AI assistant (Claude), directed by me in response to specific use cases we face in high-compliance environments. Every number was measured on real hardware, and the code is public so you can check it.

A few weeks ago TypeSafe released Jev, and it changed how I think about putting language models inside software.

Most of what we ask models to do in production isn't writing. It's deciding. Which team should get this ticket? Does this message need a human? Is this claim eligible under our policy? How urgent is this, on a scale of one to three? For years the standard answer was to prompt a general chat model, ask it to reply in JSON, parse whatever came back, and hope it didn't wander off script.

Jev skips the text entirely. You send it a piece of state and a set of typed questions (pick one, yes/no, or a score), and it sends back probabilities. No parsing, no retries, no invented options. In our tests it answered in about 135 milliseconds, over the internet.

The probabilities are the real prize. If a model tells you it is 97% sure a ticket belongs to billing, you can route that ticket automatically and send the 60% cases to a person. That pattern, automate the confident decisions and escalate the rest, is how you put AI into processes where mistakes are expensive. Routing, triage, guardrails, policy checks and agent stop conditions all become cheap, fast function calls. It opens up a lot of new work.

The catch: some of us can't call an API

I work with teams in high-compliance environments. For many of them, "just call a hosted API" isn't an option. Regulated data may not be allowed to leave a particular network, or a particular country. A new vendor can mean months of security and privacy review. Some systems run on networks with no route to the internet at all. And even where an external call is permitted, auditors want to know exactly which model made each decision, and that it won't change underneath you.

When you can't bring your data to the best model, you have to bring a good model to your data.

So we went looking for alternatives we could run on our own hardware. Within a week of Jev's release, at least seven open-source look-alikes had appeared. The question was whether any of them were good enough, and whether they would fit on the kind of modest GPU you can actually get approved: a single 8 GB consumer card.

Seven look-alikes in a week

Kev: Qwen3.5 models (0.8B, 4B, 9B) fine-tuned to read out option probabilities in one pass.
Von: a 395-million-parameter encoder built for speed.
Laya: small multilingual encoders behind a language router.
Rizzo Flow: 1.7B and 4B models served through llama.cpp, with a built-in playground.
SemIf: no training at all; it scores options straight from an ordinary model's next-token probabilities.
Nimble: a 9B model with an open data-curation and training recipe.
NanoJev: a small, fully trainable replica, released with a checkpoint trained on game states.

All of them copy the same basic idea, and four of them speak Jev's HTTP interface, which made a fair comparison possible.

How we tested

Everything ran on one NVIDIA GeForce RTX 4060 with 8,188 MiB of memory, inside a virtual machine with the GPU passed through. We ran each tool alone, one request at a time, and put every one behind the same /v1/systemone interface so a single scorer could grade them all.

For the questions we used an independent public suite: jabr/classifier-benchmark v2, 866 synthetic cases across 49 everyday decision tasks such as support routing, refund and warranty eligibility, phishing and fraud checks, triage, tone and grammar. Its cases were generated and cross-checked by a committee of seven different LLMs, and it is released into the public domain. None of the tools was built around it, and our runs reproduce its author's published scores for Jev, Von and Laya almost exactly.

Three of the tools only ship full-precision weights, which don't fit in 8 GB, so we added 4-bit and 8-bit loading to them with bitsandbytes. We also ran Jev itself through TypeSafe's API, and a plain generative model (qwen3:8b through Ollama, asked for a JSON answer) as the old-fashioned baseline. As a cross-check we ran every setup on a second suite, Kev's own transfer-v4; those results are in the repository and agree on who comes first.

What fits in 8 GB

Almost everything, once quantized. The only one that didn't fit was Nimble's 9B model, which loads at 7.5 GB before doing any work and runs out of memory on its first request.

Peak GPU memory per setup against the 8GB RTX 4060:

NanoJev 0.6B — 2.4 GB
Rizzo 1.7B · q8 — 2.5 GB
SemIf 4B · 4-bit — 3.2 GB
Von — 3.3 GB
Kev-4B · 4-bit — 3.8 GB
Rizzo 4B · q4 — 4.1 GB
Kev-0.8B — 4.5 GB
SemIf 4B · 8-bit — 5.0 GB
Kev-4B · 8-bit — 5.3 GB
Laya — 5.5 GB
Rizzo 4B · q8 — 5.7 GB
qwen3:8b via Ollama — 5.9 GB
Kev-9B · 4-bit — 7.4 GB
Nimble-9B · 4-bit — 7.5 GB idle, out of memory

Peak GPU memory while scoring, sampled every half second.

Quantization cost little. Kev-4B scored 0.878 at 8-bit and 0.872 at 4-bit, and Rizzo 0.807 and 0.799; only SemIf lost more, about three points. For a small GPU that trade is easy: Kev-4B needs more than 8 GB in full precision and runs in 3.8 GB at 4-bit.

Accuracy against speed

Results — accuracy on 866 questions against median time per question, including a local HTTP hop:

Setup	Accuracy	Wrong at ≥90% conf.	Automatable @5% err	p50 ms	Peak VRAM
Jev 1.13 (hosted)	0.968	0.2%	100%	135	hosted
Kev-9B · 4-bit	0.893	0.7%	76%	223	7.4 GB
Kev-4B · 8-bit	0.878	0.0%	77%	385	5.3 GB
Kev-4B · 4-bit	0.872	0.1%	75%	188	3.8 GB
SemIf 4B · 8-bit	0.866	3.6%	74%	168	5.0 GB
SemIf 4B · 4-bit	0.837	5.0%	66%	130	3.2 GB
qwen3:8b via Ollama	0.827	n/a	n/a	207	5.9 GB
Rizzo 4B · q8	0.807	12.4%	22%	71	5.7 GB
Rizzo 4B · q4	0.799	11.5%	3%	74	4.1 GB
Von	0.724	1.7%	21%	18	3.3 GB
Kev-0.8B	0.718	0.3%	14%	29	4.5 GB
Rizzo 1.7B · q8	0.639	22.5%	3%	35	2.5 GB
Laya	0.585	3.1%	0%	21	5.5 GB
NanoJev 0.6B	0.343	0.5%	0%	29	2.4 GB
Nimble-9B · 4-bit	did not fit	—	—	—	7.5 GB idle

Kev-4B at 4-bit is the best all-rounder. It scores 0.872, 9.6 points behind Jev's 0.968, in 3.8 GB at 188 ms per question. Its confidence is trustworthy: only 0.1% of its answers are wrong at 90% confidence or more (Jev: 0.2%), so with a 5% error budget you could automate about 75% of its decisions without review. The 8-bit build is a hair more accurate but twice as slow. Kev-9B scores a little higher (0.893), but it uses 7.4 GB of the 8 GB card, leaving no room for anything else.

SemIf is a close second with no training at all. Scoring options straight from an unmodified Qwen3.5-4B reaches 0.866 at 8-bit and 0.837 at 4-bit. That makes it the natural choice if you can't adopt a new model but can run one you already trust. Its confidence needs calibrating first: 3.6–5% of its answers are confidently wrong.

Von is the speed pick. It answers in 18 ms in 3.3 GB, ten times faster than Kev-4B, and scores 0.724. It is dependable on tone and routing but drops to coin-flip level on rule-based checks such as warranty eligibility, phishing and suspicious transactions.

The rest are harder to recommend today. The generative baseline scores a respectable 0.827 but gives no usable confidence: 17% of its answers are confidently wrong. Rizzo is fast (71 ms) and scores about 0.80, but 11–12% of its answers are confidently wrong until you calibrate it, and it struggles with policy rules. Laya (0.585) and Kev-0.8B (0.718) trail Von, and NanoJev's game-trained checkpoint scores near chance.

Jev is still the best, and a small experiment shows why

A 9.6-point gap is large, and it widens on the hardest tasks: on grammar checking Jev scores 0.81, while Kev-4B and SemIf score 0.48. But the gap is not just a score. Jev's real advantage shows up when you ask it something new.

While writing this article we tried exactly that. Ten different AI models each rewrote the article's opening paragraph to sound more natural, keeping every fact. We added two control paragraphs written as deliberately extreme styles, one stiff corporate prose and one very casual. Then we asked Kev-4B and Jev the same three questions about each paragraph: was it written by a human rather than an AI, how natural does it sound, and which of the ten sounds most human.

Rewrite by	Kev: P(human)	Jev: P(human)	Jev: natural	Jev: picked as most human
Fable 5.1	0.69	0.57	0.76	12%
Opus 5.5	0.68	0.58	0.75	39%
Sonnet 5	0.70	0.60	0.76	7%
Opus 4.6	0.70	0.56	0.75	5%
Codex (OpenAI)	0.73	0.56	0.72	3%
Haiku 4.5	0.70	0.56	0.70	2%
hermes3:8b (local)	0.71	0.45	0.67	30%
OpenCode big-pickle	0.69	0.48	0.56	2%
qwen3:8b (local)	0.69	0.47	0.43	1%
gemma4:e2b (local)	0.70	0.49	0.40	1%
Control: stiff corporate prose	0.66	0.25	0.14	—
Control: deliberately casual	0.68	0.59	0.84	—

Sorted by Jev's average rank across its three measures. "Natural" is a 3-level score scaled 0–1. "Picked" is the share of ten rotated pick-one questions; chance is 10%.

Kev could not tell. It scored the stiff and casual controls within 0.014 of each other, and its three measures disagreed on the winner. Jev separated the controls sharply (0.25 against 0.59 on the human question, 0.14 against 0.85 on naturalness), and its measures agreed with each other. Its three top picks were all rewrites that kept every fact, and it ranked the two smallest local models' rewrites last on every measure.

This is one small experiment with twelve paragraphs, not a benchmark. But it points at a real difference. Kev is very good at the decision shapes it was trained on. Jev appears to generalize further, to questions its builders probably never trained for. If you can use Jev, use Jev.

Where the alternatives earn their place

For those of us who can't, the open models are more than a consolation prize, because we control them. Four opportunities stand out.

Train on your own decisions. Kev ships its full training recipe: LoRA adapters on Qwen3.5 base models, trained on labelled examples. Its authors report the 0.8B model trains on a 4 GB GPU. A model fine-tuned on a few thousand of your own labelled decisions could close much of the gap to Jev on the questions you actually ask. We haven't done that yet; it is our next step.
Calibrate on your own data. Several tools let you fit their confidence to your labelled examples (Rizzo has a rizzo calibrate command). Calibration doesn't change which answer wins, but it is what makes a confidence threshold something you can defend to an auditor.
Pin and audit the model. A local model is a file with a checksum. It never changes unless you change it, and every decision it makes stays on your network.
Tier the decisions. Let a fast local model take the confident, routine calls and escalate the rest to a person or, where policy allows, to Jev. Because every one of these tools speaks the same interface, switching between them is a configuration change.

Others are measuring too

We aren't the only ones testing this new category, and the independent results agree on the main points: Jev leads, and the small encoders trail the 4B models.

jabr/classifier-benchmark: the public-domain suite we scored everything on. Its own results (Jev 0.964, Von 0.724, Laya 0.585) match ours.
umstek/zero-shot-ie-bench: the broadest survey, covering 38 zero-shot extraction, classification and typed-decision systems, local and hosted, in nine languages.
scienthoon/jev-ood-calibration: tests whether Jev's probabilities stay honest on a task it can't have seen. Accurate, but overconfident on rules that aren't in the text.
cacan/system1-bench: a dashboard comparing Laya and several general LLMs against Jev on support-triage cases. Some of its answer keys were aligned with Jev's own answers, so Jev's 100% there is partly by construction.
jaredpalmer/kev: Kev's frozen suites and scoring code. Our scorer's metric definitions are based on it.

What we add is the constraint: a fixed 8 GB memory budget, the larger open models quantized to fit it, and the question a compliance team actually asks, which is how much of this you can safely automate on hardware you control.

Try it yourself

The scorer, the harness, the quantization patches, the adapters, every result and the humanness experiment are at github.com/Autom8ly/gutcheck-bench. With any /v1/systemone server running, one command scores it:

python -m gutcheck.benchmark --endpoint http://127.0.0.1:8008 \
    --model kev-latest --suite suites/jabr-v2.jsonl --out runs/my-tool

If you run it on different hardware, or on a labelled set from your own domain, I'd like to hear what you find.

About this article. This article and the code in the gutcheck-bench repository were mostly generated by AI (Claude, by Anthropic), directed by me in response to specific use cases we face in high-compliance environments. The measurements are real: every number comes from runs on the hardware described above, on 25 and 26 September 2026. We are not affiliated with TypeSafe or with any of the projects tested. Our Jev runs cost a few cents in total.

How we scored. We wrote a small, tool-neutral scorer (gutcheck) so that no contestant grades itself. Its metric definitions are based on Kev's open-source scorer (Apache-2.0), and we checked that it reproduces Kev's scores exactly on every run we made with both.

Caveats. One GPU, one machine, synthetic decision tasks. Your own decisions may rank these tools differently, so test on a labelled set from your domain before choosing. Jev's latency depends on your distance from TypeSafe's servers. All seven projects were days old when tested and are changing quickly; treat these numbers as a snapshot.