decider is a language model that does not generate text. It reads a state and a set of typed questions and returns, from one forward pass, a probability distribution for every question. There is no decoding, no parsing, and no output outside the options you defined.
This post is what it does, how it is built, what the two public leaderboards say about it, and where it fails.
The interface
A typed decision is a question with a fixed answer set. Three kinds:
- Choice over 2 to 255 named options.
- Score over 2 to 10 described levels.
- Noul, the probability that the answer is yes.
You pass a state (any JSON) and a dictionary of questions. You get back, per question, the chosen option, its confidence, and the full distribution.
pip install decider-ai
from decider.infer import Decider
d = Decider("Mapika/decider-2b") # one CUDA GPU, bf16, about 4 GB
d.system_one(
{"ticket": {"messages": [{"from": "customer", "text": "I was charged twice for order A-104. Please refund the duplicate."}]},
"refund_policy": "Duplicate charges are eligible for a refund."},
{"department": {"type": "choice", "instructions": "Which team should handle this?",
"criteria": {"returns": "Exchanges, refunds, wrong or damaged items",
"billing": {"what": "Charges, invoices", "not_for": "delivery"},
"other": None}},
"refund_requested": {"type": "noul", "instructions": "Does `ticket.messages[0].text` request a refund?"},
"frustration": {"type": "score", "instructions": "How frustrated is the customer?",
"criteria": ["calm", "frustrated", "very frustrated"]}})
# department: billing 0.56 (returns 0.44, other 0.00)
# refund_requested: 0.99
# frustration: level 1 of 3, 0.55
The plain form is d.decide(state, [{"question": ..., "options": [...]}]). There is an HTTP server with the same wire format as TypeSafe's Jev API, so their SDKs work against a local decider unchanged.
How it works
Every question is rendered into the prompt with lettered options and an answer slot. The model is run once. At each answer slot the logits of the option letters are read, divided by a fitted temperature, and passed through a softmax. That is the whole readout: the distribution is the model's own next-token belief over the letters, nothing is sampled and nothing is parsed.
Because the readout is a single position in one forward pass, the cost of a decision is the cost of a prefill. On one B300 in bf16, batch of one, no CUDA graphs and no compilation:
| model | median per decision |
|---|---|
| decider-2b | 18 ms |
| decider-35b-a3b (3B active) | 41 ms |
The CUDA-graph engine the package uses by default is faster than these plain numbers.
The models
All weights are on the Hub under Apache 2.0.
| model | base | trained how |
|---|---|---|
| decider-0.8b | Qwen3.5-0.8B-Base | supervised |
| decider-2b (v10) | Qwen3.5-2B-Base | supervised, then calibration-aware RL on live browser tasks and exact games |
| decider-2b-vision | Qwen3.5-2B vision-language | supervised on image questions |
| decider-35b-a3b | Qwen3.5-35B-A3B-Base | supervised, routed experts frozen, Muon on the block matrices |
The training mixture is public datasets plus questions labelled by a local 27B teacher. Nothing was distilled from Jev. The mixture, the trainer and the evaluation code are in the repository; the recipe runs on one GPU for the small models.
A result we did not plan for: the 35B's text weights loaded onto the vision-language version of its base answer image questions zero-shot. On 71 held-out image questions from The Cauldron it picks the gold answer 91.5% of the time, against 87.3% for the purpose-trained decider-2b-vision on the same rows.
What the leaderboards say
Two third-party leaderboards rank this model class. We did not run either; the numbers are theirs.
Decision Index (edition 0.1, 22 September 2026, by multimodalart) runs every open reproduction of Jev over the same 132,422 requests on one RTX PRO 6000, with unanswered requests counted as wrong. Its score is the mean of five capability areas. decider-35b-a3b is fourth at 54.3, behind Jev at 59.5 and two zero-training wrappers on stock models at 55.7 and 55.6. The gap is knowledge and reasoning: GPQA, GSM8K, CRUXEval and MMLU.
The index also reports calibration: expected calibration error over ten confidence bins on 33 benchmarks, where confidence is the probability placed on the chosen option.
| entry | calibration error (points) | wrong at 95%+ confidence |
|---|---|---|
| decider-35b-a3b | 3.1 | 0.4% |
| jevfire (stock Qwen3.8-27B) | 6.3 | 1.5% |
| Jev 1.13.0 | 6.5 | 2.0% |
| decider-2b | 8.8 | 0.9% |
decider-35b-a3b is the best-calibrated entry of the 32, Jev included. Its mean confidence is 0.673 and its accuracy on the same answers 0.675. Calibration is a readout on the site, not part of the score; a model can be well calibrated and wrong.
JevBench (read 21 September 2026) scores four axes. decider-35b-a3b is tenth of 36 at 68.9, pulled down by the cost axis; decider-2b is twenty-first at 64.6, pulled down by calibration on the hard tier.
Limits, stated plainly
- One pass cannot do multi-step arithmetic. There is no chain of thought, so GSM8K-type items and multi-hop chains are out of reach. Split such a judgment into several questions.
- Calibration on hard items is the weak axis of the 2B. Its top-label calibration error on JevBench's hard items is 0.30. The 35B's is 0.15.
- Knowledge-heavy multiple choice. The 2B improves little over its base on MMLU; the 35B closes part of the gap at three to four times the cost.
- Games are not solved zero-shot. The released models do not play Tetris, Breakout or Space Invaders from a text state; they sit at random level. The RL stage moved probability mass on exact games, not win rates.
- Rules written into the question are not followed at this size. A convention has to be in the training data.
- English only.
The full measurement tables, including the regressions between versions, are in docs/RESULTS.md and docs/CHANGELOG.md in the repository.
What is next
The base model sets the knowledge score. Under our readout, stock Qwen3.6-27B and Gemma-4-26B-A4B-it both read above our trained 35B on a sample of the index, so the next series will start from post-trained bases, keep the readout and the calibration, and train only where training beats the stock model, which on our measurements is the 2B and 4B sizes.
- Code and docs: https://github.com/Mapika/decider
- Weights: https://huggingface.co/Mapika
- Package:
pip install decider-ai
This is an independent project, not affiliated with or endorsed by TypeSafe AI.
Top comments (0)