TL;DR
On the System One Mosaic Benchmark (S1MB) decision-engine leaderboard, our model Darwin-27B-ZTC-v2 ranks number one among 102 models, with a Borda score of 89.58 and a task average of 66.46. What makes the result different is how it decides: a single forward pass, zero generated tokens.
What S1MB tests
S1MB measures typed-decision quality. Each item gives a condition and candidate choices, and the model must pick and score the correct one. It is a decision benchmark, not a text-generation benchmark. The output is scored against a typed key, so the ranking is clean and reproducible, with no fuzzy string matching and no second judge model.
Why zero tokens wins here
Most systems that need a decision still run a full generation loop and parse the text back into a decision. That is slow, non-deterministic, and every generated token is a chance to drift. Our method, ZTC (Zero-Token Confidence), skips it: the model reads the problem in one forward pass, takes the final-layer hidden state, and applies a calibrated probe to produce the decision directly.
# one forward pass, zero generated tokens
h = model(text).last_hidden_state[last_non_pad_token]
score = ((h - mu) / sd) @ w # calibrated probe -> typed decision
Nothing is sampled, so the same input always gives the same decision, which is what you want in both a measurement and production.
The result
| Metric | Darwin-27B-ZTC-v2 |
|---|---|
| S1MB rank | Number 1 of 102 |
| Borda score | 89.58 |
| Task average | 66.46 |
Where to find it
- S1MB number one model: https://github.com/final-bench/s1mb
- ZTC method and inference code: https://github.com/final-bench/ztc
- Weights: https://huggingface.co/FINAL-Bench/Darwin-27B-ZTC-v2
FAQ
What is S1MB?
The System One Mosaic Benchmark measures typed-decision quality: given a condition and candidate choices, a model picks and scores the correct one.
Which model is number one?
Darwin-27B-ZTC-v2, with Borda 89.58 and task average 66.46 among 102 models.
What is a zero-token decision?
A decision produced in one forward pass with no generated text, by applying a calibrated probe to the model's final hidden state, which makes it deterministic.
Is it open source?
Yes, under Apache-2.0.
Top comments (0)
Some comments may only be visible to logged-in visitors. Sign in to view all comments.