I wanted a way to show that a newer model is better than an older one that wasn't a table of numbers.
Benchmark tables are boring and, worse, they're unconvincing. Nobody feels anything reading MMLU: 78.3 → 81.1. But put two models on opposite sides of a tic-tac-toe board, enforce the rules with a real engine, and let them play a best-of-five, and you get something else entirely: an outcome you genuinely don't know in advance, a scoreboard you can argue with, and a format where the audience picks a side in the comments. The progression from an older model to a newer one stops being a delta and becomes a record.
So I built Model vs Model. Two OpenAI-compatible endpoints, three games — tic-tac-toe, Connect 4, and a five-round structured debate — one three.js board that renders every move live, and a persistent scoreboard across the whole session.
The interesting engineering isn't the games. It's that models are unreliable narrators of their own actions, and the whole thing only works if you refuse to trust them.
Architecture
The shape is deliberately boring: a browser, a dev proxy, an Express server, and a provider. All the interesting decisions live inside the server box.
┌──────────────────────────────────────────────────────────────────────────┐
│ BROWSER Vite + React + TS :5173 │
│ │
│ state machine: idle ──► playing ──► finished / error │
│ three.js: one persistent scene, swaps board contents per game │
└───────────────┬───────────────────────────────────────────▲──────────────┘
│ POST /api/game │ SSE frames
│ { game, rounds, settings } │ event: start
│ │ event: move
▼ │ event: done
┌───────────────────────────────────────────────────────────┴──────────────┐
│ VITE DEV PROXY :5173 ──► :3001 /api/* (unbuffered for SSE) │
└───────────────┬───────────────────────────────────────────▲──────────────┘
│ │
▼ │
┌───────────────────────────────────────────────────────────┴──────────────┐
│ EXPRESS SERVER :3001 │
│ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ MATCH RUNNER match.ts │ │
│ │ │ │
│ │ for each game: │ │
│ │ for each turn: │ │
│ │ playTurn() ──► call model ──► parse ──► apply() │ │
│ │ ▲ │ │ │
│ │ │ rejected? │ │ │
│ │ └─── ONE retry ◄─────┘ │ │
│ │ 2nd failure = forfeit, │ │
│ │ turn passes, board untouched │ │
│ └──────┬───────────────────────┬──────────────────────┬────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌──────────────┐ ┌────────────────────┐ ┌───────────────────┐ │
│ │ RULES ENGINE │ │ LLM CLIENT │ │ JSONL LOG │ │
│ │ games/*.ts │ │ llm.ts │ │ log.ts │ │
│ │ │ │ │ │ │ │
│ │ render() │ │ plain fetch to │ │ append-only │ │
│ │ parse() │ │ /chat/completions │ │ one record/move │ │
│ │ apply() │ │ no SDK │ │ logs/matches │ │
│ │ terminal() │ │ │ │ .jsonl │ │
│ └──────────────┘ └─────────┬──────────┘ └───────────────────┘ │
└─────────────────────────────────┼────────────────────────────────────────┘
▼
┌────────────────────────┐
│ MODEL PROVIDER │
│ OpenAI-compatible │
│ POST /chat/completions│
└────────────────────────┘
The core decision: the server owns the board
A model never sees game state. It sees a string — a rendered text board — plus a move history, and it returns a move. That move is validated before it touches anything.
That constraint is what makes the whole project tractable, and it's expressed as one generic interface:
export interface GameEngine<S> {
id: GameId;
name: string;
/** Roles in play order, e.g. ['X','O']. */
roles: [string, string];
/** Max moves before the game is declared drawn (safety valve). */
maxMoves: number;
createState(seed: number): S;
/** Plain-text board for the prompt. */
render(state: S): string;
/** Compact board for the wire (client renders 3D from this). */
serialize(state: S): unknown;
systemPrompt(state: S, role: string): string;
userPrompt(state: S, role: string, ctx: TurnContext): string;
/** Strict-ish extraction of the move from raw model output. */
parse(raw: string): MoveParse;
/** Rules validation. Never mutates; returns a new state on success. */
apply(state: S, role: string, move: string): ApplyResult<S>;
terminal(state: S): TerminalResult;
/** Line shown in the move log, e.g. "X plays cell 4". */
describe(role: string, move: string): string;
}
The generic parameter S is the whole trick. runMatch() in match.ts never knows whether it's playing tic-tac-toe or running a debate. It knows how to call a model, how to retry, how to forfeit, how to log, and how to stream — and it delegates every game-specific question to the engine. Tic-tac-toe's state is (string|null)[]; Connect 4's is a 6×7 grid with a gravity rule; the debate's is a list of {role, round, text} turns. The runner doesn't care.
Two details make this hold up. First, apply() never mutates: it returns a brand-new state on success and an error on failure. So a rejected move cannot leave a half-written board behind, because there's no board to write to. Second, terminal() is a pure query, so the runner can ask "is this over?" as often as it likes without side effects.
Adding a fourth game means writing one file. Nothing in the runner changes.
The retry/forfeit state machine
This is the part I'm happiest with, and it's the part that took the most thought.
Models produce malformed output constantly. They wrap JSON in prose, they return {"move": "four"}, they play in an occupied cell, they ignore the gravity rule. A naive implementation drops the bad move, or crashes, or — worst — applies something half-parsed and corrupts the board.
The rule I settled on: one correction retry, then a forfeit. Never silently drop a move, never corrupt state.
The retry is not just "try again". It's a constrained retry that hands the model its own rejected output, the exact reason for rejection, and the list of legal moves:
for (let attempt = 1; attempt <= 2; attempt++) {
const messages: ChatMessage[] = [{ role: 'system', content: system }];
if (attempt === 1) {
messages.push({ role: 'user', content: user });
} else {
// The correction retry explains exactly what was wrong and what is allowed.
messages.push({ role: 'user', content: user });
messages.push({ role: 'assistant', content: attempts[0].raw || '(no output)' });
messages.push({
role: 'user',
content: [
`Your previous answer was REJECTED by the rules engine: ${lastError}`,
`Legal moves right now: ${legalMoves.length ? legalMoves.join(', ') : 'none'}.`,
'Reply again with ONLY the corrected JSON object. No prose, no explanation.',
].join('\n'),
});
}
// ... call the model, parse, then apply()
}
Feeding the rejected output back as an assistant turn matters. It puts the model in a conversation with its own mistake rather than asking it to try again blind, and it reliably fixes the "I wrapped the JSON in prose" class of failure.
If the second attempt also fails, the turn is a forfeit. And here's the semantic that took me a rewrite to get right: a forfeit passes the turn. It does not end the game. The board is untouched, the opponent plays next, and play continues. If a game reaches the move cap without a terminal state, the side that forfeited more turns loses — but a single forfeit is a penalty, not a loss.
Every attempt is preserved. The JSONL record for a turn keeps both attempts, so a rejected move is visible forever:
{
"moveNumber": 9, "player": "A", "model": "…", "move": "4",
"legal": true, "illegal": false, "forfeit": false,
"correctionAttempted": true, "correctionSucceeded": true,
"attempts": [
{ "attempt": 1, "raw": "{\"move\":3}", "legal": false,
"error": "column 3 is full — gravity means a disc cannot be placed there. Legal columns: 0, 1, 2, 4, 5, 6" },
{ "attempt": 2, "raw": "{\"move\":\"4\"}", "legal": true }
],
"result": "A"
}
One record per move, with the full retry trail inside it.
Gravity as a testable rule
Connect 4's rule is the best demonstration because it's a rule a model can see and still violate. The board renders as text, with a legend and a full-column marker:
0 1 2 3 4 5 6
| . | . | . | . | . | . | . |
| . | . | . | . | . | . | . |
| . | . | . | . | . | . | . |
| . | . | . | . | . | . | . |
| . | . | . | . | . | . | . |
| R | Y | R | Y | R | Y | R |
full? F
And apply() enforces it:
const row = landingRow(state, col);
if (row === -1) {
const open = Array.from({ length: COLS }, (_, c) => c).filter((c) => !columnFull(state, c));
return {
ok: false,
error: `column ${col} is full — gravity means a disc cannot be placed there. Legal columns: ${
open.length ? open.join(', ') : 'none'
}`,
};
}
const rows = state.rows.map((r) => r.slice());
rows[row][col] = role;
return { ok: true, state: { rows, moveNumber: state.moveNumber + 1 } };
That error string is not decorative. It's the exact text the correction retry hands back to the model, and it's what appears in the move log with a red badge. Because it names the rule and lists the legal alternatives, the retry almost always succeeds — which is the point. The retry exists to fix recoverable mistakes, not to punish.
To prove the path is real, the mock provider I use for testing deliberately breaks the rule: on a schedule, when a column is full, it plays into that column anyway. The retry then succeeds and the whole thing shows up in the log as a two-attempt record. I didn't want a retry path that only existed in theory.
The reasoning-token trap
Two things bit me here, and both are worth knowing.
max_tokens has a floor of 900, and it is not negotiable. A move is a few tokens — {"move": 4} is about five. So setting max_tokens: 20 seems obviously right. It isn't. If the model does any hidden reasoning, the reasoning consumes the budget, the model is cut off mid-thought, and message.content comes back empty. You get a perfectly successful HTTP 200 with nothing in it. The floor is enforced in one place:
export function effectiveMaxTokens(requested: number): number {
const n = Number.isFinite(requested) ? Math.floor(requested) : 900;
return Math.max(900, n);
}
Reasoning tokens live in a nested usage field. Not usage.reasoning_tokens on every provider, and definitely not in the message:
const reasoningTokens: number =
usage?.completion_tokens_details?.reasoning_tokens ??
usage?.reasoning_tokens ??
0;
And the rule I enforced throughout: reasoning_content is counted, never stored. The provider sends it; the client reads the usage field and discards the text. It is never logged, never put in an SSE event, never rendered. Only the token count survives, because that's the number the audience actually cares about — it's the visible cost of thinking.
I test this adversarially. The mock provider returns a long reasoning_content string on every call, and the harness asserts that the string never appears in the JSONL log or in the DOM. If someone later "helpfully" adds reasoning to the UI, the test fails.
SSE over POST
I needed the server to stream each move as it's decided. EventSource is the obvious tool and it's the wrong one: it's GET-only, and a match request carries a settings object including an API key. Putting a key in a query string is not something I'm willing to do.
So the client uses fetch with a POST body and reads the response body as a stream, buffering on the SSE frame delimiter:
const reader = res.body.getReader();
const decoder = new TextDecoder();
let buffer = '';
for (;;) {
const { value, done: streamDone } = await reader.read();
if (streamDone) break;
buffer += decoder.decode(value, { stream: true });
// SSE frames are separated by a blank line.
let sep: number;
while ((sep = buffer.indexOf('\n\n')) !== -1) {
const frame = buffer.slice(0, sep);
buffer = buffer.slice(sep + 2);
dispatch(frame, handlers);
}
}
The buffering matters: a chunk boundary can land mid-frame, so you accumulate and only dispatch on a complete \n\n-terminated frame. The frame parser also skips comment lines, which is how the 15-second heartbeat (: ping) keeps intermediaries from timing the connection out:
for (const line of frame.split('\n')) {
if (line.startsWith(':')) continue; // heartbeat
if (line.startsWith('event:')) event = line.slice(6).trim();
else if (line.startsWith('data:')) dataLines.push(line.slice(5).trim());
}
One extra thing on the server: the Vite dev proxy will happily buffer an event stream unless you tell it not to, so the proxy config forces cache-control: no-cache, no-transform on text/event-stream responses.
The three.js rendering
One persistent BoardScene owns the renderer, camera, OrbitControls, lights, and a render loop. Swapping games calls setGame(), which disposes the previous board's meshes and rebuilds — the scene itself never gets torn down. The React wrapper pushes board snapshots in; it doesn't manage three.js objects.
Tic-tac-toe pieces are extruded: the X is two crossed BoxGeometry bars, the O is a torus laid flat, both with real shadow casting and a slight emissive lift so they read against the dark plate. Winning lines glow by walking the piece's materials and pushing emissive to gold.
Connect 4 gets the animation I cared most about. Discs don't fade in — they fall, under real integration, and bounce once with damping:
// Falling discs with a single damped bounce on landing.
const gravity = -22;
for (const f of this.falling) {
if (f.bounced) continue;
f.vy += gravity * dt;
f.y += f.vy * dt;
if (f.y <= f.target) {
f.y = f.target;
if (!f.bounced && Math.abs(f.vy) > 1.2) {
f.vy = -f.vy * 0.28; // bounce
Each move also nudges the camera toward the cell that was just played, then eases back to the resting framing. It's a small offset applied to the base camera position, so it composes with whatever the user has done with OrbitControls:
nudgeToward(worldPoint: THREE.Vector3, strength = 0.22): void {
const dir = worldPoint.clone().sub(this.baseTarget);
dir.y = 0;
if (dir.lengthSq() < 1e-6) return;
dir.normalize().multiplyScalar(strength * 2.4);
this.nudge.from.copy(this.camera.position);
this.nudge.to.copy(this.basePos).add(dir).setY(this.basePos.y + strength * 1.2);
this.nudge.t = 0;
}
Testing without an API key
I didn't have a key while building this, and honestly that constraint improved the code. I wrote a small OpenAI-compatible provider (scripts/mock-provider.mjs) that reads the board out of the prompt and plays legal moves — it parses the "empty cells you may play" list, does win/block checks, and picks sensibly. It also breaks the gravity rule on a schedule, and returns reasoning_content on every call.
That gives me two harnesses that run with zero credentials:
-
npm run verify— 76 checks on the rules engine, retry/forfeit paths, and the JSONL log. -
npm run ui-check— 65 checks driving the real UI in headless Chrome over the DevTools Protocol.
The principle I held to: assert against independently re-derived state, not the server's own output. When the harness checks that a tic-tac-toe game ended legally, it doesn't trust the server's outcome field. It re-reads the final board, counts X's and O's, confirms |X − O| ≤ 1, scans all eight lines for a win, and only then checks that the server's claim matches. For Connect 4 it scans every column bottom-up to prove no disc is floating. For the log it re-reads the file and counts records per game.
This caught real bugs. A server that reports "legal terminal state" while its board is subtly wrong would pass a naive test and fail this one.
The bugs I actually hit
These are the ones worth your time.
req.on('close') killed the SSE stream after one event. The first end-to-end run hung forever: the browser got event: start and then silence. The match was running fine — records were being written to the JSONL log the whole time — but every subsequent write was suppressed. The cause: I guarded writes with a closed flag set by req.on('close'). In Node, the request stream emits close as soon as its body has been fully consumed. express.json() consumes the POST body immediately, so close fired before the first move, and every write after that was silently dropped. The fix is one word:
// Listen on the RESPONSE, not the request: `req` emits 'close' as soon as the POST body
// has been consumed, which would suppress every subsequent write.
res.on('close', () => {
closed = true;
});
Forfeit semantics: it must pass the turn, not end the game. My first implementation treated a forfeit as an immediate game loss. That's a defensible reading of "forfeit" and it's wrong for this format — it makes a single malformed response decide a match, and it makes a best-of-five meaningless when a model has one bad turn. A forfeit now passes the turn; the game continues; the move cap plus a per-seat forfeit count decides the result if no terminal state is ever reached. The harness explicitly asserts the game continued after a forced forfeit rather than aborting.
moveNumber collisions. Because a forfeited turn doesn't advance the board's own move counter, the next player's move reused the same number — two JSONL records for "move 1" in the same game. The fix was to separate the two concepts: moveNumber in the log and on the wire is a monotonic turn number, incremented on every turn including forfeits, while the board's internal counter tracks applied moves. Uniqueness is now a property the harness checks.
Records appended before the result was stamped. I wrote each record to disk as the move happened, then stamped the game's result field onto the in-memory array afterward. Append-only JSONL can't be edited, so on disk every record was missing result. The fix was to buffer a game's records in memory and append them once the outcome is known — still append-only, still one record per move, now with the result present. The tradeoff: a hard crash mid-game loses that game's records. For a game bounded at 42 moves I'll take it.
gl.readPixels returned black. My browser check read pixels straight out of the WebGL context and got zeros, which looked exactly like "the board isn't rendering." It was a false alarm: the browser clears the drawing buffer after compositing, so a read-back outside a draw call always returns black even when the frame rendered perfectly. I replaced it with a screenshot analysed pixel-by-pixel (via a small zlib-based PNG decoder), which measures what the user actually sees. I nearly "fixed" a working renderer because of this.
An unrelated process on port 3001 silently swallowed requests. Another project on my machine was listening on 3001. My Vite proxy pointed there, so requests went to that server, which returned its own health payload and 404'd the game endpoint. My first instinct was that my code was broken. Now the API port is configurable (API_PORT=3210 PORT=3210 npm run dev) and both harnesses refuse to start against an occupied port rather than reporting false passes.
An unreachable endpoint produced a slow forfeit cascade. Point the Base URL at a dead host and every call fails. My retry logic dutifully retried, forfeited, passed the turn, and did the same for the next twenty turns — eventually "completing" a match where nothing happened. A transport failure is a configuration problem, not a gameplay one, so it now aborts the match after two consecutive unreachable calls and surfaces the provider's real error text in the UI.
That last one has a general lesson: retry-and-forfeit is the right policy for bad output, and the wrong policy for broken infrastructure. Conflating them hides your own misconfiguration behind a plausible-looking game.
Playing any two models: the player library
The first version hardcoded exactly two slots: one Base URL, one key, Model A and Model B. That
is the right shape for a single comparison and the wrong shape for everything after it. You
cannot compare a hosted model against a local one, and you cannot put five models from the same
provider in a rotation without re-typing the key five times.
So the config became a two-level library:
Endpoint ── owns the base URL and the API key
▲
│ endpointId
│
ModelEntry ── owns a model name
▲
│ seatA / seatB
│
Seat
The reason for two levels rather than one flat list of "players" is the second use case in the
title of this section. If each player carried its own key, "one endpoint with five models" would
mean pasting the same secret five times, and rotating that key would mean editing five rows. Put
the key on the endpoint and the model list becomes free.
On the wire this stays simple. The client resolves a seat down to a single flat object before
starting a match, so the server never learns that a library exists:
// web/src/lib/settings.ts
export function resolveCompetitor(config: AppConfig, modelId: string): Competitor | null {
const entry = config.models.find((m) => m.id === modelId);
if (!entry) return null;
const endpoint = config.endpoints.find((e) => e.id === entry.endpointId);
if (!endpoint) return null;
return {
label: entry.label || entry.model,
baseUrl: endpoint.baseUrl,
apiKey: endpoint.apiKey,
model: entry.model,
};
}
Which means the match runner did not have to change at all. It already took a seat and a model
name; now it takes a seat and a small object that happens to include the endpoint. The generic
GameEngine<S> boundary paid for itself a second time.
Two bugs this feature introduced, both worth knowing about.
First, the API key leaked into the done event. Seats used to hold a bare model name, so
echoing seats back to the client was harmless. Once a seat held the whole competitor — key
included — that echo started shipping secrets back over the wire and into the browser. The fix
is a redaction pass, and the harness now asserts the key is absent from the payload and from
the DOM:
/** The `done` event carries the seat assignment; it must not carry the keys with it. */
function redactSeats(seats: { A: Competitor; B: Competitor }) {
return { A: redactCompetitor(seats.A), B: redactCompetitor(seats.B) };
}
Second, a misconfigured endpoint quietly forfeited an entire match. The old transport-failure
guard only fired if no call had ever succeeded:
if (transportSuccesses === 0 && transportFailures >= 2) throw new Error(...)
With one seat working and one seat broken, transportSuccesses was never zero, so the broken
seat forfeited every single turn and the match "completed" with a lopsided score. That is a
config error wearing a gameplay costume. The fix is to count consecutive failures per seat,
because a seat that cannot be reached twice in a row is not a model playing badly — it is a
wrong URL or a bad key:
if (turn.transportError) {
transportFailures[seat]++;
if (transportFailures[seat] >= 2) {
const target = seats[seat];
throw new Error(
`Could not reach ${target.label || target.model} at ${target.baseUrl} — ` +
`two consecutive calls failed. Check that endpoint's base URL and API key in Players.`,
);
}
} else {
transportFailures[seat] = 0;
}
The general lesson, and it is the same one as before: retry-and-forfeit is the right policy for
bad output and the wrong policy for broken infrastructure. Adding a second endpoint made the
distinction load-bearing, because now one seat can be perfectly healthy while the other is
unreachable.
The library also has to keep the app always playable. Deleting an endpoint cascades to its
models, and a seat pointing at a deleted model is repaired rather than left dangling; the final
endpoint and the last two models cannot be deleted at all. The browser harness walks that whole
path — add a second endpoint, add a third model, attach it, delete the endpoint, confirm both the
cascade and the seat repair — because "the user can break their own config" is exactly the kind
of thing that only shows up in a real browser.
What I'd build next
- A fourth game to prove the engine abstraction. Nim or Reversi would each be one file.
- ELO across sessions, persisted outside the JSONL log so the scoreboard survives a restart and accumulates across days.
- A judge model for the debate. Right now the debate is scored by a deterministic published rubric (cap compliance, engagement, concreteness, concision) so results are reproducible. A judge model would be more interesting and less honest — worth building as an explicit alternative with the bias caveats visible.
- Replay from JSONL. Every record has the seed, the move, and the board. A scrubbable replay of a past match is nearly free.
-
Cost and token accounting per match, since I'm already logging
prompt_tokensandcompletion_tokens. - Parallel matches on different seeds, to turn a single game into a distribution — which is what you actually need before claiming a model is better.
- A head-to-head ladder where models play every game in rotation, so the first-move advantage averages out across a schedule instead of being neutralised one game at a time.
Caveat
I built and verified this without an API key. Everything you can test locally is real: the rules engines, the retry and forfeit paths, the SSE streaming, the JSONL log, the three.js board, the two harnesses — 76 rules checks and 65 browser checks, all passing against a local OpenAI-compatible mock.
What is not verified is the request shape against the specific provider it defaults to. The defaults point at https://api.particle.ai/v1 with models deepseek-v4-flash-0731 and deepseek-v4.1-flash, but I never sent a live request to that endpoint. The client uses response_format: { type: "json_object" }, a max_tokens floor of 900, and optionally chat_template_kwargs: { "enable_thinking": false } when the "Disable reasoning" toggle is on — and whether that last field is accepted, ignored, or rejected by a given provider is exactly the kind of thing you only learn by trying it. If the provider refuses the request, the real error text appears in the UI banner rather than a generic message, which is the next best thing to having tested it.
Code & more: https://www.dailybuild.xyz/project/257-head-to-head-games
Top comments (0)