We made language models play chess against each other — live, around the clock

Evgenii — Wed, 23 Sep 2026 08:50:38 +0000

▶ Watch the live stream: https://www.youtube.com/live/sKpIL4CpUXw

Every move on our stream is a real, live request to a language model: Gemini, Claude, GPT, Grok, Llama, Qwen, Mistral, DeepSeek and more. No engines, no tools, no pre-recorded games. The broadcast runs on YouTube almost non-stop — https://www.youtube.com/live/sKpIL4CpUXw — and all standings and games are on kremen-arena.ru.

The first tournament is over, the second one is live right now, and it turned out far more interesting than we expected.

What you see on the stream

The board, both clocks, and a live win-probability bar computed by Stockfish.
"Thinking": a short excerpt of what the model wrote before its move.
A counter of moves the model did not get right on the first try — you can see at a glance who "sees" the board and who is guessing.
A passport for each model: vendor, release date, Elo rating.
A LIVE badge in the corner at all times — and REPLAY when we show recorded games — so you always know whether you are watching a game as it happens.

Between games: the result, the standings and the next pairing. When a tournament ends: the podium, congratulations, and a countdown to the next one.

The rules: same for everyone

For the results to mean anything, every model plays under identical conditions:

one system prompt for everyone (its hash is published), temperature 0, one request per move, no tools or external engines;
60 seconds per move; 20 minutes per side plus 5 seconds per move, and the clock is spent by the model's real response time;
an illegal move costs time and triggers a retry that includes the list of legal moves; after three misses a random legal move is played, and five of those in one game is a forfeit;
provider errors (overload, network) are never the model's fault: the request is retried for free, without touching the clock;
the model version and the provider are pinned for the whole event;
pairings come from the hash of a Bitcoin block whose number is announced in advance, so anyone can recompute them.

Chess Machines Cup I: the results

Ten models, round robin, 45 games: 19 ended in checkmate, 15 in a draw by the rules, 9 on time, 2 by forfeit.

Place	Model	Points	Illegal on 1st try
1	Gemini 3.6 Flash	9 / 9	0%
2	Claude Sonnet 5	7	19%
3	Llama 3.1 8B Instruct	5.5	72%
4	Grok 4.3	5	4%
5–6	Cohere Command R7B, GPT-5.1	4.5	92% / 9%
7	Amazon Nova Micro	3.5	77%
8	Mistral Nemo	3	67%
9	Microsoft Phi-4	2	54%
10	Qwen3.7 Flash	1	37%

What surprised us:

Gemini 3.6 Flash won every single game — and never played an illegal move on the first try.
Bronze went to Llama 3.1 8B, a small model from 2024. Almost three quarters of its first attempts are illegal, but once it gets the legal-move list it plays stubborn chess: five draws and a single loss, ahead of GPT-5.1 and Grok 4.3.
Qwen3.7 Flash lost on time in a position Stockfish gave it 93% to win. It kept thinking at its usual pace until the flag fell. That game made us add a time-trouble rule: with under 15 seconds on the clock, each reply gets 5.

What building it taught us

A few things we learned the hard way, in public, during the first live week:

Output budget plus prompt must fit the provider's context window. One model's endpoint had a 16,384-token window; we reserved 16,000 tokens for the answer. Short prompts fit — until the model made an illegal move late in the game and the retry prompt, now carrying the legal-move list, grew past ~390 tokens. Every retry was rejected, and the game had to be annulled. The fix was trivial; noticing it took three voided games.
A model served by a single provider is a liability on a live stream. When that one provider rate-limits, the game stalls in front of viewers. We now only admit models that several providers serve.
Validate the pinned provider before the season, not during it. When we set up the next two cups, 14 of the 20 new participants had a provider slug that simply did not serve that model. Our pre-flight check now compares every participant against the public OpenRouter catalog — model exists, provider serves it, context window fits — before a single paid request is made.

What's next

The plan for this week (Moscow time, UTC+3):

Wed–Thu — Chess Machines Cup II and III, ten new models each (DeepSeek, GLM, Gemma, Nemotron, Qwen, Ling, MiMo and others). Cup II is live now;
Fri — Grand Prix: the podium finishers of all three cups plus the highest-rated fourth-place finisher;
Sat — the final four;
Sun — an underdogs tournament;
Mon — the best games of the week as replays, clearly marked REPLAY.

How it works

A Django + Celery backend runs the games, python-chess enforces the rules, Stockfish evaluates positions, and model requests go through OpenRouter with the provider pinned. The stream is produced on the same server without OBS: a browser overlay page in Xvfb, Chromium and ffmpeg at 720p.

Come watch — and tell us in the comments which model should play in the next cup.

▶ Live stream: https://www.youtube.com/live/sKpIL4CpUXw · standings and every game: https://kremen-arena.ru