DEV Community

Cover image for How I Built a Computer Vision Chess Board Detector
MD ABUBAKAR
MD ABUBAKAR

Posted on

How I Built a Computer Vision Chess Board Detector

#ai #machinelearning #webdev #showdev

I Built a Chess Scanner That Converts Any Chess Image Into a FEN + Analyzes Games Like Chess.com

👉 Live website: https://chessscanner.seedrlite.in/

🧠 Introduction

Chess engines are extremely powerful — but only when they have a FEN position or a PGN to analyze.
Most of the time, people see a board in a photo, a screenshot, or a video frame… and wish they could instantly analyze it.

So I built a web application called Chess Scanner that:

✅ Detects the chessboard from any image
✅ Identifies all 12 piece types (White/Black × King/Queen/Rook/Bishop/Knight/Pawn)
✅ Generates a perfect FEN string
✅ Lets you edit the detected board
✅ Analyzes PGNs and classifies moves (Good, Great, Brilliant, Miss, Blunder, etc.)
✅ Plays best moves using Stockfish

This post explains how I built it, the challenges I faced, and how it works under the hood.

♟ 1. The Idea

My goal was simple but ambitious:

“Upload any chess image → get instant FEN → analyze it like on chess.com.”

There are apps that do individual tasks, but none that combine:

  • Computer vision
  • FEN extraction
  • Real-time board editing
  • Game analysis
  • Chess.com-like move classification

I wanted this all in one clean web app.

🏗 2. Architecture Overview 

Frontend (React + Tailwind)
        ↓ Sends image
Backend (FastAPI + YOLOv8)
        ↓ Detects board + pieces
Board Processor → Generates FEN
Stockfish Engine → Analysis
Frontend Visualization → Move types + best moves
Enter fullscreen mode Exit fullscreen mode

Tech Stack

  • React + Tailwind for UI
  • FastAPI (Python) for backend
  • YOLOv8 (Ultralytics) for board & piece detection
  • Custom dataset (Roboflow)
  • Stockfish 17.1 for move evaluation
  • SQLite authentication system
  • JWT auth
  • Cloud VPS (Ubuntu, Nginx)

🔍 3. Training the Vision Model

Dataset

I created my own labeled dataset using Roboflow:

  • Board Detection Model

    • Segmentation + bounding box
    • Trained on real + synthetic chessboard images

  • Piece Detection Model

    • 12 classes (white/black for each piece)
    • Augmented with flip/rotate/lighting variations
    • Special training on dark boards, unconventional themes, and low-light images

Challenges

  • Some chess sets have similar shapes (e.g., black pawns vs black bishops in bad lighting).
  • Perspective distortion makes squares uneven.
  • Large boards in mobile screenshots require high-resolution inference.

Results

My best YOLOv8 model achieves:

  • mAP50: ~0.98
  • mAP50-95: ~0.84
  • High accuracy even with dark backgrounds

🧩 4. Converting Detection Output to FEN

This is the magical part.

Once pieces are detected:

  1. The board segmentation mask is used to warp the image into a perfect 8×8 grid.
  2. Each square is assigned a piece (if detection center falls inside).
  3. Empty-square compression creates the FEN row string.
  4. The final FEN is returned to frontend.

Example output:

7N/5p2/1p3P2/p7/kp6/1p6/1P6/1K6 w - - 0 1
Enter fullscreen mode Exit fullscreen mode

You can then analyze or edit the board directly.

♜ 5. PGN Analyzer + Move Classification

Another major feature I added:

✔ Analyze any PGN

I process each move with Stockfish and compare:

  • Eval before
  • Eval after
  • Best engine move
  • Centipawn loss
  • Sacrifices
  • Move difficulty

Then I classify moves in categories inspired by Chess.com:

  • Brilliant
  • Great Move
  • Good Move
  • Book Move
  • Inaccuracy
  • Mistake
  • Blunder
  • Miss (Missed tactic)

This gives players a clean, human-friendly understanding of their game quality.

🎨 6. UI/UX

The UI focuses on clarity and speed:

  • Drop zone for uploading images
  • Real-time board preview
  • FEN editor
  • Move list with evaluation graph
  • Best move suggestions
  • Dark/Light themes
  • Full mobile support

I also added a Chess Scanner logo featuring a board + camera icon.

🚀 7. Deployment

Technical deployment stack:

  • Ubuntu 22.04 VPS
  • Nginx → Gunicorn → FastAPI
  • Project served as /api/infer endpoint
  • React frontend deployed with Nginx static hosting
  • SSL via Certbot
  • YOLO model files optimized with half precision

Everything runs smoothly even on a mid-tier VPS.

📈 8. What’s Next?

Future improvements:

🔹 Real-time video scanning
🔹 Automatic PGN generation from detected positions
🔹 Opening explorer integration
🔹 ELO estimation based on move accuracy
🔹 Cloud account system + saved games
🔹 Mobile app (React Native, already under development)

🔗 Live App: Chess Scanner

🏁 Conclusion

Building a chess scanner that works reliably on ANY image was a challenging but rewarding project.

The combination of:

  • YOLOv8
  • Smart board geometry
  • Stockfish evaluation
  • React UX

makes the tool extremely powerful for:

  • Chess learners
  • Coaches
  • Content creators
  • Daily puzzle solvers

If you want to try it or want the repo, feel free to reach out.

Top comments (0)