You know the moment. Someone posts a chess puzzle as a screenshot, or you hit a diagram in a chess book, and you just stare at it, calculating in your head, because the position is trapped inside a picture.
I built fenshot to fix that: paste any chessboard image, get the position as a FEN, open it on a real board. It runs entirely in the browser, nothing gets uploaded, and the recognition model is 1.3 MB. This post is about the part that took the most iterations: teaching a tiny CNN to read boards from any site, any theme, and any chess book, without hand-labeling a single tile.
The existing open source model didn't cut it
The classic solution is tensorflow_chessbot, which has been around for years and works fine on the themes it was trained on. But it was trained on a narrow set of board styles. In practice that meant:
- queen/king confusions on chess.com themes
- basically useless on book diagrams
- up to 34 wrong tiles per board on my eval screenshots
34 wrong tiles out of 64 is not a recognition error, it's a different position. A tool that silently hands you the wrong FEN is worse than no tool.
The trick: labels are free if you render the data
Hand-labeling chess tiles is miserable work. But chess has a property most vision problems don't: you can render a known position, and then every one of the 64 tile labels is true by construction.
So the entire training set is synthetic. The real work moves from labeling to a different question: how do I make renders look like actual screenshots people paste?
The corpus generator:
- pulls ~72 complete piece sets and ~55 board textures (lichess assets, plus others used strictly as training input)
- renders positions from 50% random chess.js playouts (realistic piece distributions) and 50% uniform random placements (class balance for rare pieces; a tile classifier doesn't care about legality)
- adds procedural flat two-color boards with random color pairs, so it generalizes to any site's flat theme, plus hatched boards biased toward print-style piece sets, which is what makes chess books work
- draws the decorations real screenshots have: last-move highlights, arrows, coordinate labels
Then it degrades everything the way the real world does: JPEG artifacts from quality 35 to 95, blur, dimming overlays (the Reddit lightbox case), resize round-trips, and ±3px corner jitter to simulate the board detector being slightly off.
That last one matters more than it looks. The model that ships never sees perfectly cropped tiles in production, so it should never see them in training either.
Zero train/serve skew
Training tiles are extracted through the exact same code path that runs in the browser: same grayscale conversion, same board extraction, same tile slicing. Not a reimplementation of it in Python, the literal same TypeScript functions.
This sounds like a detail. It is the whole ballgame. Most of the mysterious "works in training, fails in prod" gaps in small vision models are the two pipelines disagreeing about something boring like rounding during resize.
The model is tiny on purpose
class TileNet(nn.Module):
def __init__(self):
self.conv1 = nn.Conv2d(1, 32, 3, padding=1)
self.conv2 = nn.Conv2d(32, 64, 3, padding=1)
self.conv3 = nn.Conv2d(64, 64, 3, padding=1)
self.fc1 = nn.Linear(64 * 4 * 4, 256)
self.fc2 = nn.Linear(256, 13)
Three conv layers, two dense, ~330k parameters, 1.3 MB as fp32 ONNX. Input is a 32x32 grayscale tile, output is a softmax over 13 classes (empty + 6 pieces x 2 colors). Trains in 8 epochs on a MacBook.
It runs in the browser via onnxruntime-web on WASM. Classifying all 64 tiles takes tens of milliseconds. There is no server, no API, no per-call cost. The model ships with the app like a font file.
The classifier was the easy half
The part nobody warns you about: finding the board in a page-wide screenshot is harder than reading it. The detector does gradient peak analysis to find the grid, and then a stack of recovery logic earned one failure case at a time: reconstructing a square board from a single clean axis, arbitration between candidate grid alignments (classify both, higher mean confidence wins), an empty-board rescan, and one-tile parity repair.
And one rule I'd recommend for any recognition tool: an honesty contract. Every read carries per-tile confidence, and there's a plausibility gate on top (a "board" with no kings on it is a false positive, so it gets masked and rescanned). When the tool isn't sure, it says so instead of returning a confident wrong answer. For a chess tool this is existential: a user who gets one silently wrong position never trusts another read.
Results
On the real-screenshot eval set (the acceptance gate runs the full production pipeline, not the model in isolation):
- legacy model: up to 34 wrong tiles per board, book diagrams unreadable
- shipping model: zero wrong tiles on every positive case, and images with no board still correctly rejected
Everything is open
The whole thing is MIT: the recognition engine (@scoriiu/fenshot on npm), the trained model, and, the part I care most about, the full training pipeline (asset fetcher, corpus generator, training script, eval gate). The model is reproducible, not just downloadable.
- Try it: fenshot.com (paste a screenshot, ctrl+v)
- Browser extension (reads the board on any page in one click): Chrome / Firefox / Edge
- Code: github.com/scoriiu/fenshot
If it ever reads a board wrong, send me the picture. That's how the detector got every one of its recovery tricks.
Top comments (0)