Two local decision models play chess, and every probability is on screen

Neither one generates text. Neither one searches. Both are 300 to 800 million parameters, both run on this laptop, and every move either one considers is drawn on the board with the probability it gave it. This is the story of building that, and of the three bugs that only showed up once other people ran it.

---

The idea in one line

Put two decision models on a chessboard, print the probability each one gave every move it was offered, and then score both of them against a ground truth derived by exhaustive search.

First, what a decision model is

A decision model is not a chat model with JSON mode. You give it a state and a question whose answer type is fixed in advance, and it returns a value your code can use directly: one option from a set you supplied, a probability that a statement is true, or a level on an ordered scale. One forward pass. Nothing is generated. No tokens come out, so there is nothing to parse and nothing to validate.

out = model.decide(
    state_text,
    {"move": {"type": "choice",
              "instructions": "Which move should the side to move play?",
              "criteria": {"Kc7": "king d8 to c7", "Rc6": "rook f6 to c6"}},
     "mate1": {"type": "boolean",
               "instructions": "Does the side to move have a forced checkmate in one move?"},
     "depth": {"type": "score",
               "instructions": "How many of the side to move's own moves are needed to force checkmate?",
               "criteria": ["no forced checkmate within two moves",
                            "checkmate in one move",
                            "checkmate in two moves"]}})

out["answers"]["move"]["choice"]          # the move it picked
out["answers"]["move"]["probs"]           # calibrated, sums to 1
out["answers"]["move"]["prediction_set"]  # conformal set at the configured alpha
out["answers"]["move"]["abstain"]         # True: not confident enough to act on
out["latency_ms"]                         # all three questions, one read of the state

Three things make that useful rather than cute:

• The option set is yours. The model cannot invent a move you did not offer it, so the failure mode is a wrong choice, never a malformed one.
• The probability is calibrated, and the model ships a conformal set and an abstain flag. So the interesting question is not only "was it right" but "did it know".
• The cost is one forward pass. Latency is tens to hundreds of milliseconds and the output token count is zero.

And one thing makes it useless on its own: it judges the state in front of it. It cannot see that a locally neutral move sets up a good one three moves later.

Why chess is the right test

That last property is exactly what chess can measure, because chess contains both cases:

task	what it needs	fair?
mate in one	read the board, see the check	yes. One ply, fully determined by the state.
mate in two	look one move ahead for the opponent's reply	no. This is where a state-only judge should fail.
no mate inside two	see that nothing forces	the control, and it gives the boolean both of its classes

So the experiment is not "can a small model play chess". It is "where exactly does judgment stop working and search become necessary", with a number attached.

Watching two of them play

The benchmark is the measurement. The game is the demo, and it is the part people actually watch, so it reports the decision rather than the score:

• the position, with an arrow for the last move
• which model moved, by name, and the moves it was offered
• how many legal moves existed, and how many were offered
• its calibrated probability on every option, as bars
• its confidence, whether it abstained, and where that flag came from
• the conformal set, for the model that produces one
• the exact state text it read

Two properties are what make the game honest. A model never plays an illegal move, because it can only choose from the options it is handed and the options are legal. And the game is not scored, because a live position has no verified answer, so there is nothing to score it against. The benchmark is where accuracy lives. The game is where you watch the choices.

Neither model has lookahead, so neither will play well. That is not a bug in the harness and it is not a reason not to watch. A model that hangs a piece with 0.31 confidence is telling you something. A model that finds the only saving move with 0.05 confidence is telling you something else. Both are visible, per move.

The app needed rules, so I wrote a chess engine

The first version only measured, and measuring never needs a move generator: every board it draws is a FEN a build script computed. A game is different. It generates positions nobody has seen, and a model can only be offered legal moves if something can work out what the legal moves are.

The tempting answer is python-chess, which is already a build dependency. I did not take it. python-chess is GPL-3.0 and this app is MIT, and shipping it inside the application would change the licence of the whole thing. So app/chess_rules.py is written from scratch: the board as a list of 64 squares, pseudo-legal generation, legality by making the move and asking whether your own king is attacked, castling rights, en passant, promotion, SAN with disambiguation, and the draw conditions.

A hand-written rules engine is a liability unless it is proved, and it can be proved. Perft counts the nodes at a depth, and a move generator that gets the count right on kiwipete and the four other standard positions is a move generator that is right about castling, en passant, promotion and pins:

PERFT_POSITIONS = {
    "start": ("rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1", [20, 400, 8902]),
    "kiwipete": ("r3k2r/p1ppqpb1/bn2pnp1/3PN3/1p2P3/2N2Q1p/PPPBBPPP/R3K2R w KQkq - 0 1", [48, 2039]),
    ...
}

On top of the counts, the test plays random games and compares every legal move, every SAN string and the whole FEN against python-chess, position by position, over more than four hundred positions.

It found a bug immediately, which is the point. The first version missed queens entirely in attack detection: the sliding-ray check looked for b and r, and a queen is q, so every queen check was invisible. Perft caught it on the first run.

Ground truth by exhaustive search, not by engine evaluation

There is no Stockfish, no evaluation function, no heuristic and no neural network anywhere in the ground truth. Three definitions, all exhaustive:

def mate_in_one_moves(board):
    found = []
    for move in board.legal_moves:
        board.push(move)
        try:
            if board.is_checkmate():
                found.append(move)
        finally:
            board.pop()
    return found


def forced_mate_in_two_moves(board, stop_at=2):
    found = []
    for move in board.legal_moves:
        board.push(move)
        try:
            if board.is_checkmate():
                continue                      # that is depth 1, not depth 2
            replies = list(board.legal_moves)
            if not replies:
                continue                      # stalemate is not mate
            ok = True
            for reply in replies:
                board.push(reply)
                try:
                    if board.is_checkmate() or not has_mate_in_one(board):
                        ok = False                # we are mated, or no mate available
                finally:
                    board.pop()
                if not ok:
                    break
            if ok:
                found.append(move)
                if len(found) >= stop_at:
                    break                         # we only need none, one, or more than one
        finally:
            board.pop()
    return found

Depths are counted in the side to move's own moves, not plies, because that is the unit a player thinks in. stop_at bounds the work: we only ever need to know "none", "exactly one" or "more than one".

Cap the option set, and say so

A position can have over two hundred legal moves. No model answers a 200-option question in one pass, so the set is capped, and the rule is stated rather than hidden.

In the benchmark the cap is twelve, the key move is always included, the rest are legal moves drawn at random from a seeded generator, and the set is sorted by UCI so the answer is not always the first option.

In a game the cap is different, because there is no key move to guarantee: every capture, check, promotion and castle is kept first, then quiet moves fill the rest at random. And the cap is per model, because the models can read different amounts. Vega's window is 73,728 tokens, so 40 options fit. Laya's is 1,024 tokens with 256 of them reserved for the question and its options, so a Laya choice question is capped at 20 and the app will not offer more. The move record says which cap was used, so a capped choice is never presented as a free choice among everything.

The benchmark cap is the main limit of the experiment, and it is stated in the artifact, not just in a blog post: the distractors are random legal moves, which makes the task easier than chess is. The honest upgrade is to draw the distractors from an engine's top moves, and that is the first item on the list of things worth adding.

One instruction, identical on every set

MOVE_INSTRUCTIONS = "Which move should the side to move play? Choose the strongest move available."

Word for word the same on the mate-in-one set, the mate-in-two set and the quiet set. If it said "which move mates" on one set and not the others, the accuracy difference between the sets would be measuring the question rather than the model.

The boolean and score questions do name mate, which creates a second, smaller experiment inside the first: ask all three in one call and the mate wording may help the move question; ask them in three separate calls and it cannot. Both arms are reported, and the difference between them is a finding rather than a nuisance to hide.

The answer must not be in the prompt

An early version keyed each option by its SAN, which meant an option could be labelled Ra8#. The option text is part of the model's input, so that label hands over the answer. Now:

• the option key is SAN with + and # stripped (Ra8),
• the option description is neutral geometry ("rook a1 to a8"),
• the full SAN with its marks is kept as san_full and only shown in the UI after the decision has been made.

There is a test for this, because it is exactly the kind of leak that comes back:

def test_option_labels_never_leak_check_or_mate():
    board = chess.Board(BACK_RANK_MATE)
    for move in board.legal_moves:
        assert "#" not in chess_gt.option_label(board, move)
        assert "#" not in chess_gt.san_of(board, move)

Nothing downloads itself, and nothing leaves the machine

Both models are local, and that is enforced rather than promised.

A model is a declaration. app/models.py holds one entry per checkpoint: the repository, the licence, the size, the token window, the option limit, and the files that must exist for the model to count as present. Nothing else in the app is allowed to hardcode a repository name or a size. Vega is 1.83 GB, a 63 MB decision engine plus a 1.77 GB frozen backbone, and the allow-list skips the 4B model in the same repository because this app never loads it. Laya is 678 MB in one bundle.

Each model needs its own recorded consent before anything is fetched, and the consent file names the repository and the expected size. A vegaml.load() or a laya.Router() handed a Hub id would fetch that in the background, so neither ever is. Both loaders are handed a directory that already exists:

checkpoint = self.checkpoint_dir()
if not checkpoint:
    raise RuntimeError(
        f"{self.spec.name} is not on this machine. Open the Setup panel and "
        f"download it, or point --model-path {self.spec.key}=/path at an existing copy."
    )

model = vegaml.load(str(checkpoint), **kwargs)          # Vega
router = Router(models={key: str(checkpoint)}, ...)     # Laya

There is no stub, no mock and no cloud fallback in the shipped app. If a model is missing, the run and game endpoints answer 409 and name the model that is missing. There is no API key anywhere in the app, no .env file, and no browser storage use, because there is nothing to authenticate to.

One more line is worth pointing at. Laya's Router seeds three Hugging Face checkpoints in its constructor, so a routing decision could reach the network even though the app never asked it to. The app drops them immediately:

router.models = {key: str(checkpoint)}   # local only, nothing to route to

That single line is also half of the bug in the next section, which is why it is worth reading twice.

Two models, and the three places they differ

Vega and Laya answer the same typed questions, so they drop into the same decision interface. They are not identical, and every difference the app has to handle is declared in one file rather than scattered through a branch somewhere:

• A boolean is called two things. Vega answers with a boolean field. Laya calls that type noul and answers with a noul field. The app asks for a boolean and reads either.
• Score criteria are a list. Laya requires level descriptions as a list, lowest first. The app builds a list, so both accept it.
• Abstention belongs to one of them. Vega reports an abstain flag and a conformal set at a configured risk. Laya reports neither, so the app applies a confidence floor of 0.5 to the top probability and records that it did, with the source, in the move record. An abstain rate for Laya is the app's rule, not the model's judgement, and the record says so.

One normalisation decision matters for the comparison. The app records the top probability as the confidence for both models, so the calibration curve means the same thing across runs, and it keeps Laya's own confidence (how concentrated the distribution is) and answer_confidence (the calibrated one) under their own names.

The bug that took a reader's report: a 400, a vanishing message, and an alias

I shipped a build with both models in it. A reader downloaded Laya, pressed Load into memory, and got this:

POST /api/model/load HTTP/1.1" 400 -
Failed to load resource: the server responded with a status of 400 (Bad Request)

Two of those three lines were my fault, and neither was the actual bug.

The 400 was a lie. My error contract mapped ValueError to a 400, on the theory that a malformed request is the caller's problem. A load failure is not a malformed request. It is a fault, and the class of the exception says nothing about who caused it. The load route now answers 409 when the checkpoint is simply not on the machine, and 500 carrying the traceback for anything the loader raises, and it prints the same traceback to the terminal. A bare status code is not an error message.

The message vanished before it could be read. The setup card wrote the failure into the DOM and then called refreshStatus(), which re-rendered the cards from scratch and took the message with it. Failures are now recorded in state and re-rendered with the card, so they stay on screen until the next successful action.

The actual bug was a name. With the error finally visible, the message was:

ValueError: unknown model 'laya'; choose one of ['laya'] (or an alias: [...])

The model is unknown, and the list of available models contains laya. That is the signature of a namespace collision. Laya's alias table maps laya to its own English checkpoint, and a name is canonicalised through that table before it is looked up. So registering a checkpoint as laya registered it as english. Then the line that keeps the app local-only, the one from the last section, removed the only thing english could have resolved to. Two of my own lines fought each other, and the winner produced a message that contradicted itself.

The fix is a separate key inside the loader (laya-chess), a check that the name survives canonicalisation, and a test that fails if a future package version claims that name too.

How it was found. Not by reading the code. I installed the package, which resolved transformers to 5.20.1, and noticed that the checkpoint's small files are 35 MB against the weights' 643 MB. So I downloaded everything except the weights, handed the loader that directory, and watched which stage failed first. It got all the way to the weight check: tokenizer, encoder config, model build and the weight file all passed on transformers 5.20.1. The only thing that failed was the name.

The lesson is not "test your code". It is that an error message which contradicts itself is worth reading twice, and that a reproducible failure is worth the 35 MB it costs to reproduce it.

The bug the UI harness caught

Two panels in this app render the same thing: the play tab's move panel and the games review's move panel. Both wrote their probability bars into a placeholder with the same id. That is invalid HTML, and in jsdom it is quietly fatal: with two elements sharing an id, a scoped container.querySelector("#ply-bars") returned null, so the bars rendered in one panel and silently not in the other.

The fix is a class instead of an id. The lesson is narrower than "ids must be unique", which everyone knows: the browser tolerates the duplicate, so the bug only appears in one of the two places, and only if you look at both.

Earlier, the same harness caught a board that drew the position and no move on it, because it rendered before the position files had loaded. A board that looks fine and says nothing is exactly the class of bug a diff review cannot see.

The architecture

        data/positions/*.jsonl   (verified ground truth, sha256 in manifest.json)
                   |
                   v
  +------------------------ app/ -------------------------------------------+
  |  chess_rules.py    the rules: move generation, legality, SAN, mate      |
  |  game.py           players, the game loop, option selection             |
  |  game_store.py     append-only game logs, one JSON when a game ends     |
  |  models.py         the registry: both checkpoints, declared once        |
  |  model_manager.py  per-model consent, download, load, family adapters   |
  |  positions.py      load benchmark sets, check hashes against the manifest |
  |  state.py          render the FEN into the state text the model reads   |
  |  questions.py      build the three typed questions                      |
  |  decisions.py      call the model, score the answer                     |
  |  metrics.py        accuracy, regret, ECE, AUC, abstain split, latency   |
  |  run_store.py      append-only .jsonl while running, one .json at the end |
  |  report.py         markdown report, disagreements first                 |
  |  server.py         stdlib http.server, JSON API plus static files       |
  +-------------------------------------------------------------------------+
                   |
                   v
  web/   plain DOM, hand-written SVG, no framework, no bundler, no CDN

Server side is Python standard library only. The UI is plain HTML, CSS and DOM with hand-written SVG for the board and the charts, which means there is no build step, no dependency to audit, and nothing loaded from a CDN.

The run pipeline

POST /api/run/start
      |
      v
  model present? sets loadable?   --no--> 409 with the reason
      |
      v
  append-only <run_id>.jsonl created
      |
      v
  worker thread: model.load()   (never downloads)
      |
      +--> per position:
      |      state.py      -> state text
      |      questions.py  -> one sheet or three sheets
      |      model.decide  -> probabilities, conformal set, abstain
      |      decisions.py  -> correct? regret? probability vs key?
      |      run_store.py  -> append immediately
      |
      v
  metrics + markdown report, disagreements first

The append-only log matters more than it looks: a run that crashes halfway keeps every decision it already made, and the report is built from what is on disk rather than from what is in memory. The same shape is used for games, so a game that dies on move forty keeps its thirty-nine moves.

The board, which is the part people actually look at

A screenshot is not reviewable in a pull request and I cannot see my own render, so the board is checked structurally instead. Pieces, coordinates and arrows are drawn in a single SVG layer whose viewBox is 8x8 with one unit per square, which means no fixed font sizes anywhere and no pixel maths:

// one unit is one square, so an arrow is just two centres and a shortened line
const from = squareCenter(arrow.from);   // {x: file + 0.5, y: 8 - rank + 0.5}
const to = squareCenter(arrow.to);
const ux = (to.x - from.x) / length, uy = (to.y - from.y) / length;
const endX = to.x - ux * 0.36, endY = to.y - uy * 0.36;

Pieces use the solid unicode shapes for both colours, filled and stroked in CSS. The outline glyphs look thin and washed out at board size; a solid shape with a contrasting edge reads from across a room.

The numbers worth reading

Accuracy is the least interesting thing a run produces.

metric	what it answers
accuracy vs tie-aware chance	did it pick a mating move, against what guessing scores
mean regret, in moves	0 found the fastest mate, 1 found a slower mate, 2+ found no mate inside two
calibration error	is a 0.9 right nine times in ten
abstain split	accuracy when it answered against accuracy when it refused
conformal coverage	does the prediction set hold the answer at the configured alpha
latency p50 and p95	wall clock per position, every call the mode needed

Regret is the one to quote, because it converts a pass or fail into a cost:

scored["regret"] = max(0, option["depth"] - row["best_depth"])

And the abstain split is the one that tells you whether the confidence machinery is worth anything. If accuracy when it abstains is not lower than accuracy when it answers, the flag is decoration. That test matters more now that there are two models, because only one of them produces the flag itself.

Verifying it, because you should not take my word for it

pip install python-chess pytest

python scripts/verify_positions.py --deep-options   # every label, option and line ply
python -m pytest tests -q                           # rules, game, adapters, scoring, shipped data
node tests/board_render.mjs                         # what the board draws
npm install jsdom --no-save && node tests/ui_harness.mjs

The verifier recomputes mate in one, mate in two, and the depth of every single option from the FEN, walks every forced line ply by ply, and fails on any disagreement. It also checks each file's SHA-256 against the manifest, so a hand edit shows up as changed in the UI.

The Lichess sets carry a second, independent label: the puzzle's own published solution. The fetcher records whether the search agreed with it and prints the count, because a ground truth nobody cross-checks is just an opinion with better formatting.

The UI checks drive the real page in jsdom rather than a mock: start a game, watch the board draw, read the move table, step the review ply by ply, and now also fail a load on purpose and check that the error is still on screen after the re-render. That last one exists because a message that vanished is what turned a five minute fix into a support thread.

What is not claimed

• No results ship with the repo. The run file is the source of truth, and the results page is a template.
• Option sets in the benchmark are capped at twelve legal moves, which is a gift to the model.
• Neither model has lookahead, so neither plays well. The games are a demonstration, not a strength claim.
• Laya has no abstain flag of its own, so its abstain rate is the app's confidence floor, and the record says which rule was used.
• A mate in one is a one-ply question. One ply is not chess.
• The generated demo positions are not human games. The Lichess sets are.
• Every rate is a count over the positions in the run. Read the n.

Fork it

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python -m app            # UI on http://127.0.0.1:8787

Setup has one card per model and asks for consent before it downloads anything. Play puts any model against any other, or against random legal moves. Benchmark runs either model on the same verified positions and compares them side by side. docs/FORKING.md covers the row schema, adding a model, adding a question, adding a set and changing a metric.

The takeaway

A decision model is a different instrument from a chat model, and the way to find out what it is good for is to build a task where the right answer is computable and the wrong answer is visible. Chess gave me both: a ground truth you can derive by exhaustive search, and a board that makes the mistake obvious to anyone watching, including me.

The part I did not expect is how much of the work is in the reporting. Two models that disagree on the same position, each with a probability attached, is a more useful artifact than either model's accuracy, and it only exists because the choices were drawn on the screen.

Video Walkthrough

Screenshots

Code & more: https://www.dailybuild.xyz/project/285-vega-laya-chess