The web is bloated with Jev posts. The thing is a few weeks old, and my feeds are already full of it.. benchmarks, hot takes, threads about decision models being the future of agents. Jev, if you've managed to avoid the hype, is TypeSafe AI's hosted decision API where you send a situation and a few typed questions, it returns the answer with a probability for every option.
Two spoilers, because I like roads with signposts. Classical ML never left the chat, and the newest tool in the box still doesn't answer the oldest question, whether you actually need it in the first place.
At some point I stopped skimming news and got really curious about the machinery underneath. How does a model decide instead of write? What is actually different in there?
The short answer turned into a project, and the project turned into a story about building a thing and then discovering, layer by layer (sorry), that most of it already existed. It's called take-my-layer.
Init
I'll be honest, I never thought I'd invented something. I knew reading probabilities off logits was a thing and the Jev posts themselves hint at it. But knowing a trick exists and holding it in your own hands are different things, and what I didn't know was how far it stretches. I took a 4-bit Qwen model on a laptop, no training, no labels, questions made up on the spot. That part was worth testing.
So I went scouting before building anything serious. Kind of following my own advice from a post about fog of war as a mental model. The fog doesn't mean the territory is missing, it just means you haven't been there yet. Reading papers is scouting. Building a small proof of concept is scouting. I expected to find maybe one or two adjacent things.
Well.. the territory I found was huuuge :D
The trick is from 2021
First stop. Reading answer probabilities instead of generating answer text. This is how the whole evaluation world has scored decoder-only models for years.. it's the loglikelihood default in lm-evaluation-harness. The prompting version has names (verbalizers, pattern-verbalizer pairs, cloze-style prompting). PET (Schick & Schütze, EACL 2021) and "Small Language Models Are Also Few-Shot Learners" (NAACL 2021) are the classics, LM-BFF (ACL 2021) even automates the label-word search.
What I'd been calling "reading the logits" has a standard name, a literature, a codified abstraction, and several thousand citations. The trick predates ChatGPT (obvious for people who are not the first day on the ML street).
The bugs are from 2021 too
The naive version has a bug called position bias. Ask "A, B, C or D?" and the model leans toward whichever option is listed first, regardless of the options. I found that myself the fun way when the raw read on my coding data predicted bash for 74 of 120 states. A bash magnet kind of. Its 0.57 accuracy was roughly bash's base rate in the data.
Essentially every bug I hit someone had already hit:
- Calibrate Before Use (Zhao et al., ICML 2021): content-free priors to correct the model's answer skew.
- Order sensitivity of option lists (Pezeshkpour & Hruschka, 2023).
- Fantastically Ordered Prompts (Lu et al., ACL 2022): same disease, few-shot demonstrations.
- PriDe (ICLR 2024): the cheap permutation-plus-prior combination, which is more or less the cheaper cousin of my scheme.
- Batch Calibration (ICLR 2024): the unified view of all of the above.
My own fix (arrived at independently and with more cursing), was to list the options in several rotated orders, ask once per order, average. That's called permutation marginalization, and it is worth +4.7 points of agreement with the reference over the naive single-prompt read (5 answers flip toward the reference, 0 away). All of that was known. Standing on shoulders of giants, etc.
Leaving early
The next piece was speed. A full forward pass over a 4B model for a one-word answer is wasteful, and it turns out the decision signal sits pretty early. (Qwen3.5-4B has 32 blocks; layer selection kept picking 16 and 19, before the last full-attention blocks. The interesting states live in the linear-attention trunk.)
The literature here is CALM (NeurIPS 2022), early exit per token by confidence. In my case it's a fixed half-depth exit for one fitted head, the layer-pruning family. Fine again, keep standing on the shoulders!
In MLX land there is exactly one static precedent that I could find: NeoJev, a 27B 4-bit (~15 GB) that keeps 56 of 64 layers with a self-distilled head, 0.727 mean accuracy at 310 ms p50 on an M5 Pro. One detail I genuinely enjoyed is that a plain logit lens at their layer 56 reads 0.663. Same head-over-raw gap I measured on my own side, so kind of proving I'm moving in the right direction.
A practical note for anyone following this route: mlx-lm exposes no intermediate-layer hooks (issue #1053, still open as of date of this blog post), so a hand-rolled block loop is the only way in. Mine matches the stock forward to a max logit difference of about 0.037.
A warning from the probing people
Fitting a tiny linear head on hidden states is also a literature: called probing. Hewitt & Liang (2019) and Belinkov's survey (2022) are the standard references, and their criticism is aimed at claims. A probe is weak evidence about what a model "understands". I took that as a warning about wording but not about architecture. A production head is judged by held-out task accuracy on the served artifact (ours reads 0.826 on its holdout), not by what it supposedly reveals. Linear (as the probing people say) is the defensible capacity choice.
And finally products
The method hunt was humbling, in that almost everything was already there before. The product hunt was even more humbling.
Jev you've already met. Cloudflare's Clef ships fine-tuned decision models with open Apache-2.0 weights (209 ms median, 38.8 ms for the flash variant). Unsloth has a train-your-own recipe: LoRA plus a Clef-design head, their 0.8B reads 36% on typed-decisions before training and 73% after. Around those there's a whole cluster: Mini-Jev, mlx-decision, laya-mlx. Even the routing people are adjacent, though RouteLLM and friends put a separate model in front of the LLM, whereas I wanted the served model's own distribution to be the classifier.
(!) And then the one that mattered to me: AnyJev, Nokia Applied Research's open method and library (technical report). Essentially I've taken AnyJev's method, deployed on the Mac.
What's actually mine
An honest ledger, because I care about this part. The repo's docs have a section literally titled "honest lineage", and it says explicitly that I did not invent the trick. What's original though:
- The missing MLX backend for AnyJev's toolbox (an original implementation of their documented backend contract; they never shipped one).
- The small-memory end: a 4-bit 4B, about 3 GB resident, versus NeoJev's 15.
- Zero-training deployment: the frozen model works the moment it downloads. No dataset, no training run, no labels.
- The observe() loop: while a shadow proxy watches real agent traffic, it quietly banks "what I predicted" next to "what actually happened", and one command refits a head from that bank. As far as I could find, nobody ships fit-on-your-traffic heads.
- And the measurements of course.
None of that is a new mechanism. It's just glue and a scorecard. BTW! I wrote about taste a while ago and there (TLDR) I say that when production becomes abundant, judgment becomes scarce. The judgment here was mostly deciding what not to claim.
Benchmarks
The repo holds itself to wording rules that I want to keep in this post too: fidelity claims say "aligned with the reference", never "more correct"; accuracy claims name the arena and the model size. So, with that:
| what was measured | result |
|---|---|
| Agreement with the real Jev API, 300 non-coding questions | 4B 75.7%, 9B 82.0%; Jev agrees with itself 98.7%, which is the ceiling |
| Readout vs greedy decoding of the same model, same 300 questions | +4.7 (4B), +6.3 (9B), McNemar p = 0.003; on long inputs greedy Choice answers collapse (46% vs 85% agreement) |
| BANKING77, 10 intents, gold labels | 87.5% (4B), 91.25% (9B), Jev 95.5%: a trained model wins on its home turf |
| typed-decisions, zero training | 61.6% (4B), 60.4% (9B), vs 73-78% for models fine-tuned on it |
| Latency, long input, 5 questions in one request | 22.2 s down to 3.7 s mean (the state is read once); the coding head answers in ~0.2 s |
The losses are the point, in a way. A trained model beats a zero-training readout whenever labels and a fixed label set exist. Train a classifier! This is for questions and options you cannot enumerate ahead of time, on a machine where the data cannot leave.
Here's one real exchange on my M5:
curl -s localhost:8095/v1/systemone -d '{"state": "Ticket #4821 from Dana (Pro plan, customer for 14 months): \"Since this morning every export to CSV fails with a 502. We run payroll from those exports at 3pm today.\" Status page: all green.",
"questions": {
"route": {"type": "Choice", "instructions": "Which team should take this first?", "options": ["billing", "platform-oncall", "account-management", "docs"]},
"escalate_now": {"type": "Noul", "instructions": "Does this need the on-call engineer right now?"},
"urgency": {"type": "Score", "instructions": "How urgent is this ticket?", "levels": ["low", "medium", "high", "critical"]}}}'
-> {"model": "mlx-community/Qwen3.5-4B-MLX-4bit",
"answers": {
"route": {"answer": "platform-oncall", "confidence": 0.8516, "probabilities": {"billing": 0.0744, "platform-oncall": 0.8516, "account-management": 0.0579, "docs": 0.0161}},
"escalate_now": {"answer": true, "confidence": 0.637, "probabilities": {"Yes": 0.637, "No": 0.363}},
"urgency": {"answer": "critical", "confidence": 0.6651, "probabilities": {"low": 0.0051, "medium": 0.0156, "high": 0.3142, "critical": 0.6651}}}}
Status: research prototype, and the docs are blunt about it. The Jev-agreement numbers use LLM-authored questions with no human labels. The one fitted head that ships cannot be reproduced from the repo, because its training seed is withheld. If you have a fixed label set and training data, train a classifier instead. All of it is written down in a LIMITATIONS.md.
The boring lesson
This field is moving embarrassingly fast.
But the lesson I'd actually keep from these weeks is a boring one: classical ML never left. Every number in the table above says so. Wherever a fixed label set and training data exist, something trained the old way wins. Jev's 95.5% on BANKING77 against my 87.5%, the fine-tuned 2B sitting 16 points above my zero-training readout on typed-decisions. Reading logits doesn't replace training. It covers the corner cases training can't reach e.g. questions you can't enumerate ahead of time, no labels to collect, data that isn't allowed to leave the machine etc.
So before throwing this at a task, the checklist is the one classical ML wrote decades ago. Fixed labels and enough data? Train a classifier, it's still unbeatable there. Data that can't leave the machine, or questions that change per request? That's where reading the model's own probabilities earns its keep. In between, renting a hosted API is probably a fine deal.
Oh right, the question
Ah, I actually never answered the "how does a model decide instead of write" part from the beginning. Short version, and experienced ML people will roll their eyes at how obvious it is: the decision seems to form well before the end of the network. My tiny head reads block 16 of 32 and still answers the "what next" question at 0.826 accuracy. AnyJev's own measurements found a middle block to be a better feature space than the final one, since the last blocks are busy converting the answer into token space. And NeoJev keeps 56 of 64 layers, which suggests there is slack even higher up. So roughly: the deciding happens in the first half to two thirds of the layers, and the rest is more like the writing department.
Am I even right here? Probably, for these models, by my measurements. That's as far as I'd push it.
That's it.
Top comments (0)