30 of 37 games ended in repetition: what small LLMs do in chess when they don't know what to do

Evgenii — Wed, 23 Sep 2026 19:25:50 +0000

We run a live stream where language models play chess against each other — every move is a real API call, no engines, no tools (first write-up here). Yesterday's cup had frontier-ish models (Gemini, Claude, GPT, Grok). Today's Chess Machines Cup II was ten smaller and cheaper models — and it played very differently.

▶ Watch live: https://www.youtube.com/live/sKpIL4CpUXw

Cup II results

#	Model	Points (of 9)
1	DeepSeek V4 Flash (0423)	6
2	Ministral 3B	5.5
3	Gemma 4 31B	5.5
4	Mistral Small 3.2 24B	5.5
5	Solar Pro 4	5
6–8	Hy3, Nemotron 3.5 Lightning, DeepSeek V4.1 Flash	4.5
9	Granite 4.2 8B	4
10	GLM 5.3 Flash	0 (disqualified)

Cup I vs Cup II in numbers

	Cup I (bigger models)	Cup II (smaller models)
Decisive games	67%	19%
Checkmates	19	4
Draws by repetition	15	30
Average centipawn loss per move	236	321
Blunders (≥3 pawns lost)	27% of moves	36% of moves
Illegal move on first try	42%	53%

(Average centipawn loss = how much worse, by Stockfish, a move was than the best move. Lower is better. Gemini 3.6 Flash scored 37 in Cup I; the best model of Cup II, Gemma 4 31B, scored 247 — roughly the level of Cup I's middle of the table.)

What we learned

1. When a weak model has no plan, it shuffles. 30 of 37 games ended in threefold repetition. Pieces go back and forth — a knight to f3 and back to g1, a rook along the same rank. It's not a bug in our engine; it's what the models do. A 3-billion-parameter model finished second largely by drawing.

2. Points and quality don't always agree. In Cup I, GPT-5.1 had the second-best move quality of the whole field and still finished sixth — five of its games were drawn by repetition. We're giving it the wildcard into Friday's Grand Prix.

3. "Thinking" models can be too slow for a clock. We give 60 seconds per move. Some reasoning models sat at the limit on almost every middlegame move — one was disqualified after five moves in a single game without an answer. Reasoning-effort settings and even a token budget were ignored by some providers; only turning reasoning off completely made them answer in 1–2 seconds. Surprisingly, their move quality stayed mid-table.

4. So we added a "kick". From Cup III on, a model gets 45 seconds to think. If it hasn't moved by then, the same attempt is immediately re-sent with reasoning disabled, so the move still lands within 60 seconds. It is not counted as a failure. Thinkers keep thinking when they can afford it — and nobody hangs the stream for three minutes.

5. Single-provider models are a liability. Three games today were annulled because one inference provider rate-limited us. The rules say provider errors never count against a model, so these games are simply replayed — but for a live show, redundancy matters.

What's next

Thu, 10:00 Moscow (07:00 UTC) — Chess Machines Cup III (DeepSeek, GPT-6 Luna, Gemma 4, Ling 3.0, MiMo, Qwen)
Fri — Grand Prix: the podium finishers of all three cups + GPT-5.1
Sat — Final Four · Sun — Underdogs · Mon — best games in replay

Between cups the stream loops the best games of the week, clearly marked REPLAY, with a countdown to the next live cup.

▶ https://www.youtube.com/live/sKpIL4CpUXw — and tell us which model should play next.

We made language models play chess against each other — live, around the clock

Evgenii — Wed, 23 Sep 2026 08:50:38 +0000

▶ Watch the live stream: https://www.youtube.com/live/sKpIL4CpUXw

Every move on our stream is a real, live request to a language model: Gemini, Claude, GPT, Grok, Llama, Qwen, Mistral, DeepSeek and more. No engines, no tools, no pre-recorded games. The broadcast runs on YouTube almost non-stop — https://www.youtube.com/live/sKpIL4CpUXw — and all standings and games are on kremen-arena.ru.

The first tournament is over, the second one is live right now, and it turned out far more interesting than we expected.

What you see on the stream

The board, both clocks, and a live win-probability bar computed by Stockfish.
"Thinking": a short excerpt of what the model wrote before its move.
A counter of moves the model did not get right on the first try — you can see at a glance who "sees" the board and who is guessing.
A passport for each model: vendor, release date, Elo rating.
A LIVE badge in the corner at all times — and REPLAY when we show recorded games — so you always know whether you are watching a game as it happens.

Between games: the result, the standings and the next pairing. When a tournament ends: the podium, congratulations, and a countdown to the next one.

The rules: same for everyone

For the results to mean anything, every model plays under identical conditions:

one system prompt for everyone (its hash is published), temperature 0, one request per move, no tools or external engines;
60 seconds per move; 20 minutes per side plus 5 seconds per move, and the clock is spent by the model's real response time;
an illegal move costs time and triggers a retry that includes the list of legal moves; after three misses a random legal move is played, and five of those in one game is a forfeit;
provider errors (overload, network) are never the model's fault: the request is retried for free, without touching the clock;
the model version and the provider are pinned for the whole event;
pairings come from the hash of a Bitcoin block whose number is announced in advance, so anyone can recompute them.

Chess Machines Cup I: the results

Ten models, round robin, 45 games: 19 ended in checkmate, 15 in a draw by the rules, 9 on time, 2 by forfeit.

Place	Model	Points	Illegal on 1st try
1	Gemini 3.6 Flash	9 / 9	0%
2	Claude Sonnet 5	7	19%
3	Llama 3.1 8B Instruct	5.5	72%
4	Grok 4.3	5	4%
5–6	Cohere Command R7B, GPT-5.1	4.5	92% / 9%
7	Amazon Nova Micro	3.5	77%
8	Mistral Nemo	3	67%
9	Microsoft Phi-4	2	54%
10	Qwen3.7 Flash	1	37%

What surprised us:

Gemini 3.6 Flash won every single game — and never played an illegal move on the first try.
Bronze went to Llama 3.1 8B, a small model from 2024. Almost three quarters of its first attempts are illegal, but once it gets the legal-move list it plays stubborn chess: five draws and a single loss, ahead of GPT-5.1 and Grok 4.3.
Qwen3.7 Flash lost on time in a position Stockfish gave it 93% to win. It kept thinking at its usual pace until the flag fell. That game made us add a time-trouble rule: with under 15 seconds on the clock, each reply gets 5.

What building it taught us

A few things we learned the hard way, in public, during the first live week:

Output budget plus prompt must fit the provider's context window. One model's endpoint had a 16,384-token window; we reserved 16,000 tokens for the answer. Short prompts fit — until the model made an illegal move late in the game and the retry prompt, now carrying the legal-move list, grew past ~390 tokens. Every retry was rejected, and the game had to be annulled. The fix was trivial; noticing it took three voided games.
A model served by a single provider is a liability on a live stream. When that one provider rate-limits, the game stalls in front of viewers. We now only admit models that several providers serve.
Validate the pinned provider before the season, not during it. When we set up the next two cups, 14 of the 20 new participants had a provider slug that simply did not serve that model. Our pre-flight check now compares every participant against the public OpenRouter catalog — model exists, provider serves it, context window fits — before a single paid request is made.

What's next

The plan for this week :

Wed–Thu — Chess Machines Cup II and III, ten new models each (DeepSeek, GLM, Gemma, Nemotron, Qwen, Ling, MiMo and others). Cup II is live now;
Fri — Grand Prix: the podium finishers of all three cups plus the highest-rated fourth-place finisher;
Sat — the final four;
Sun — an underdogs tournament;
Mon — the best games of the week as replays, clearly marked REPLAY.

How it works

A Django + Celery backend runs the games, python-chess enforces the rules, Stockfish evaluates positions, and model requests go through OpenRouter with the provider pinned. The stream is produced on the same server without OBS: a browser overlay page in Xvfb, Chromium and ffmpeg at 720p.

Come watch — and tell us in the comments which model should play in the next cup.

▶ Live stream: https://www.youtube.com/live/sKpIL4CpUXw

DEV Community: Evgenii

30 of 37 games ended in repetition: what small LLMs do in chess when they don't know what to do

Cup II results

Cup I vs Cup II in numbers

What we learned

What's next

We made language models play chess against each other — live, around the clock

What you see on the stream

The rules: same for everyone

Chess Machines Cup I: the results

What building it taught us

What's next

How it works