TypeSafe Jev Played Chess — And Landed Next to Reasoning Models

TypeSafe's Jev is an odd one. It isn't a chat model - you send a state plus typed questions (Choice / Score / Noul) and get labels with probabilities — like a classifier, except the input can be free text and you choose the labels each time.

I've been running models through LLM Chess for a while. LLMs play in an agentic setting with a multi-turn protocol: get_current_board, get_legal_moves, make_move <uci>. Jev doesn't do that. Can you still put it on the same leaderboard?

Apparently yes — if you talk to it the way it expects.

A note on (anti-)benchmarks

TypeSafe is openly anti-benchmaxxing. In the System One / Jev launch post they say existing LLM benchmarks don't really apply here, and they skip the usual public-eval scoreboard putting it's model next to other models. They prefer dated internal snapshots instead.

I put Jev on LLM Chess anyway — same harness, same opponents as the chat models.

Talking to Hawking

It reminded me of talking to Stephen Hawking: smart, but no free speech. Answers do not come out in a free from

Outside chess I tried a silly next-character loop — Choice over a–z plus <eos>, one API call per letter. Asked to spell blue or finish hel…, it spat out almost-words (bue, helhoh). Started from pari… and it finished paris. Generating from scratch was weak; finishing a partial worked better. A fixed list of legal chess moves fits this model much better than spelling English letter by letter.

How I wired it into LLM Chess

Each ply:

• state: { fen, side_to_move }
• one Choice: all legal UCI moves as options (SAN in the criteria text), max 255

Jev returns a UCI string. It turns into make_move e7e5 — same action chat models send to the eval runner, on the outside seems almost like normal eval interaction.

Pricing: $0.042 / 1M input tokens, output free.

Results

On the current leaderboard jev-latest is around #59, Elo ~243, sitting next to qwen3.6-27b and o4-mini-medium:

#	Player	Elo	Game Duration	Tokens	Cost/Game	Time/Game	Games
58	qwen3.6-27b@q4_k_s	270.0±148.6	100%	5956	$1.46	7.54h	27
59	jev-latest	242.9±117.5	100%	119	$0.0015	35.6s	80
60	o4-mini-medium	240.3±64.3	100%	2156	$1.11	41.5m	199

• Elo which I trat as a proxy to reasoning capability, is in the same band as mid-pack reasoning / strong small models
• Game duration 100% — full games, not early deaths on illegal moves or broken dialogs, perfect protocol following (that's what type sefity buys you).
• ~36s / game and ~$0.0015 / game (~$0.00002 / move); 80 games ≈ $0.12 total vs a dollar-plus per game for Elo neighbors. That's super fast and extermely cheap.

Draws vs Dragon

Dragon (white) vs Jev (black), 10 games per level:

Dragon	Dragon wins	Draws	Jev wins
L1	5	5	0
L2	5	5	0
L3	5	5	0

50% draws at L1, L2, and L3. Flat. With most chat models, stronger Dragon usually means fewer draws — the engine finishes more games. Here the draw rate doesn't drop across three levels. That helps the Elo.

System One on the board

TypeSafe calls these System One models: fast decisions, no deep thinking.

Chess still needs real choices — what to capture, when to trade, when to aim for a draw. Jev has no chat, costs fractions of a cent per game, and still lands near mid-pack reasoning models on Elo. So the skill isn't "writing about chess." It's picking a move from a list.

Takeaways

TypeSafe doesn't want a public leaderboard as their proof. This is one outside eval.

• Jev fits tasks where you already have a list of options (tools, routes, labels, moves). Asking it to write English letter-by-letter is a bad fit (bue / helhoh).
• ~$0.0015 and ~36s per game is a different cost/speed class than dollar-scale LLM evals.
• Useful where the product needs which option, not a paragraph — routing, structured agents, decision steps in code.

Leaderboard: LLM Chess

Harness: maxim-saplin/llm_chess (feat/typesafe-jev-player branch / TypeSafe Jev player)