DEV Community: Nicholas

Building a Chess Engine & Book Scraper - Part 4: Capping the Catalog & 4,500+ Games Cooked

Nicholas — Sat, 18 Jul 2026 15:45:38 +0000

"The data pipeline is perfectly stable, the automation is running smoothly, but I'm not pushing these changes to GitHub yet." 😅

Hey guys ;)

I’m a bit tired after this week's data marathon, so I'll let the numbers and the architecture do the talking today.

Over the last seven days, my workstation has been split into two universes: generating high-quality training datasets for my chess AI on a strict daily quota, and building a scalable market analysis tool for the book industry.

Here is the weekly recap, some crucial optimizations, and the roadmap ahead.

♟️ Chess Database: 4,500+ Games Cooked

The automated parallel self-play setup I described in my previous post is performing beautifully. My hardware survived the daily quota of 716 games per day.

The Weekly Total: 6 * 716 games + 260 baseline test games = 4,556 games successfully generated and saved into the local database.

The dataset is expanding exactly as planned, laying down a bulletproof foundation before I open the Jira backlog for the Reinforcement Learning model training phase.

📖 The Book Scraper: Smart Capping & Strategy

While the chess engine runs its daily simulations, I’ve been heavily focusing on my new target platform: Yakaboo (the largest book platform in Ukraine). The platform contains 16 distinct main categories.

Initially, I went deep and scraped over 410 pages for the Fiction category alone. But looking at the data, I realized I needed to optimize my traversal mechanics. I decided to introduce a strict limit: a maximum of 200 pages per category.

Why cap the dataset at 200 pages? 🤔

Online retail storefronts naturally sort their catalog by popularity, search volume, and sales velocity on the initial pages. The deeper you go into page 300+, the higher the density of out-of-print titles, placeholders, or niche products. Scraping thousands of deep pages won't help us identify active consumer trends-it just adds unnecessary noise and processing bloat to the database.

Here is a sneak peek of how the raw item structure looks right now:

    {
        "title": "Generations in the World of Work",
        "author": "Корі Сіміллер",
        "price": "2947 грн",
        "rating": "0",
        "reviews": "0",
        "status": "Фізична книга",
        "href": "https://www.yakaboo.ua/ua/generations-in-the-world-of-work.html"
    },
    {
        "title": "Paralegal Career For Dummies",
        "author": "Ліза Зіммер Хатч",
        "price": "1832 грн",
        "rating": "0",
        "reviews": "0",
        "status": "Фізична книга",
        "href": "https://www.yakaboo.ua/ua/paralegal-career-for-dummies-3295617.html"
    },
    {
        "title": "Performance Appraisals & Phrases For Dummies",
        "author": "Кен Ллойд",
        "price": "1513 грн",
        "rating": "0",
        "reviews": "0",
        "status": "Фізична книга",
        "href": "https://www.yakaboo.ua/ua/performance-appraisals-phrases-for-dummies.html"
    },
    {
        "title": "A Pluralistic Approach to Leadership: Interdisciplinary Perspectives",
        "author": "Натан В. Хартер",
        "price": "3982 грн",
        "rating": "0",
        "reviews": "0",
        "status": "Фізична книга",
        "href": "https://www.yakaboo.ua/ua/a-pluralistic-approach-to-leadership-interdisciplinary-perspectives.html"
    },

And of course some screenshots

It's a chess database fragment. Yep some games is short, and some is more then 100 moves.

I parsed Yakaboo for 3 days and already parsed almost 23MB of data.

My laptop straggling to parse data while I'm writing posts.

And what I see as a progress. Of course I could just open Chrome and watch how pages scrolling.

Building a Chess Engine in Python - Part 3: Letting the DB Cook & Launching a New Book Market Scraper

Nicholas — Thu, 16 Jul 2026 11:29:29 +0000

"What do you do when your laptop is too weak to train a neural network and run game simulations at the same time? I'm letting the database cook, and starting a new small project."

Hey guys ;)

Sorry for being a bit late with this update, but I promise I have some very exciting news.

Right now, my main Chess Project is entering a transition phase. To train a proper Reinforcement Learning model, I need a massive, high-quality dataset of self-play games. But running heavy simulations and coding at the same time is a heavy burden for my hardware.

So, I made a strategic decision: take a 2-week development break to generate the database, while kicking off a brand new Data Science project on the side.

Here is what's happening on both fronts!

♟️ Chess Project: The Daily Generation Quota

To keep my laptop healthy while still building a solid dataset, I’ve set up a strict automated daily schedule. From now until July 27th, I am running my parallelized self-play script with a specific daily quota:

716 games per day in total (distributed as 358 games per active CPU core).
The depth distribution: 300 games at depth 2, 55 games at depth 3, and 3 games at depth 4.

Even this "limited" run takes about 2 hours and 30 minutes every single day. Once the database reaches its target size on July 27th, I will officially close the generation phase and jump straight into the model training backlog, which is already fully planned and ready in my Jira!

📖 New Project: Analyzing the Book Industry

While the chess script is running in the background, I decided to tackle a question that has been on my mind for a while: Are people actually reading fewer books nowadays?

To answer this with cold, hard data, I decided to build a custom web scraper to gather, clean, and analyze retail book market trends.

The Pivot: From Amazon to Yakaboo

Initially, my first target was Amazon. However, I quickly ran into two major roadblocks:

Aggressive Anti-Bot Protection: Amazon does not like scrapers, making automated extraction highly inefficient without expensive proxy setups. Yep I parsed some book data from Amazon via undetected-chromedriver, but there is second problem.
Data Limitations: The accessible, free-tier product details were too sparse for the deep analysis I wanted to conduct. To compare on Yakaboo there is a books cards with title, author, rating and count of reviews, of course price. So for my analysis I only need to get amount of pages(for example via GoogleBooks API or just right from Yakaboo details page) and interest in certain book(I'm planning to get it via Google Trends)

So, I pivoted! My new target is Yakaboo, the largest online book platform in Ukraine. It has incredibly rich metadata regarding which is exactly what I need. And of course their web source code is pretty easier to read.

I’m currently writing the automation scripts combining Selenium and BeautifulSoup to handle paginated lists and extract the raw dataset.

What's Next?

For the next two weeks:

My CPU will be sweating to generate those 716 chess games daily.
I will be finalizing the Yakaboo parser, cleaning the scraped book data, and preparing my first analytical dashboard.

I’m super excited to share actual data insights about what people are reading (and buying) in one of the next post!

Have you ever had to pause a heavy computational task to prevent your machine from melting? What’s your go-to tool for bypassing anti-bot systems? Let’s share some stories in the comments!

#python #datascience #webscraping #chess #showdev

Building a Chess Engine in Python - Part 2: Achieving 100% Correctness & Setting Up Multicore Self-Play

Nicholas — Sat, 11 Jul 2026 18:18:27 +0000

It’s PEAK.

Hey guys ;)

Since my last post, I’ve done a massive amount of under-the-hood work. Honestly, writing the code this time felt surprisingly smooth, but the results are a huge milestone for the project: I built a performance/correctness tester, implemented a parallelized self-play system using multiple CPU cores, and hooked it up to a database.

We are officially at the doorstep of Reinforcement Learning. Let’s break down how it all came together.

The Heartbreak (and Triumph) of Perft Testing

In my previous update, I mentioned that I was moving toward Perft (Performance Test) to verify the engine's correctness. A Perft test recursively walks the move tree to a specific depth, counts all total leaf nodes, and compares that number to standard chess data.

When I ran my first test to depth 3 and 4, my numbers were lower than the official benchmarks on chessprogramming.org. I'm not going to lie - I was really sad. After days of wrestling with bitwise masks, realizing your engine is missing thousands of moves feels brutal.

But instead of giving up, I used the test to isolate the discrepancies. It turned out the missing paths weren't deep architectural flaws, but rather sneaky edge cases in pawn logic:

The En Passant Ghost: A bug where specific en passant capture rights weren't being cleared or generated correctly.
Out of Index Crash: A rare boundary issue when pawns reached the absolute edges of the board during specific masking checks.

Once those pawn bugs were squashed, I ran the numbers again. Success! The engine officially proves its 100% mathematical correctness up to depth 4 across multiple complex custom test positions.

Multicore Self-Play & Epsilon-Greedy Exploration

With a bulletproof bitboard engine, the next challenge was generating data for our future Reinforcement Learning (RL) agent. To train a model, you need games—thousands of them.

To achieve this, I implemented a Self-Play system. Here is how the architecture works right now:

Epsilon-Greedy Minimax: I modified my classic Minimax algorithm into an epsilon-greedy version.
Opening Imitation: To prevent the engine from playing the exact same game over and over, the first 4 moves of every match are completely random. This effectively imitates traditional chess openings and forces the engine to explore a massive variety of board states.
Multicore Scaling: Running games sequentially takes ages. I utilized Python's multiprocessing features to distribute the self-play simulation across multiple CPU cores simultaneously.
Database Storage: Every single generated game, position, evaluated score, and move state is now being piped and safely recorded into a local database.

Next Stop: Reinforcement Learning

The infrastructure is ready. The engine is fast, verified as correct, scales across my CPU, and logs clean training data directly to a DB.

The next station is implementing and training the actual RL models. This is where the real experiment begins.

#python #chess #performance #multiprocessing #showdev

Building a Chess Engine in Python - Part 1: The Immortal King & Bitboard Grid Synchronization

Nicholas — Wed, 08 Jul 2026 15:40:12 +0000

"I spent days writing logic masks, only to realize my Chess King became an immortal god who could capture the opposing King."

Connecting a classic 8x8 Pygame matrix UI to a new, high-performance bitboard engine felt like a great idea until the synchronization bugs started crawling out. Over the last few days, I've been hunting down some of the most hilarious and stubborn logical errors I've encountered so far.

Let's dive into what went wrong and how I fixed it.

1. The Mirror Dimension (`[row]` vs `[7 - row]`)

After building the adapter bridge, the standard pieces moved fine, but two visual elements broke entirely: move history highlights and available move hint dots. They were rendering perfectly on the wrong side of the board.

The Cause: A classic coordinate mapping issue. While matrices count rows from top to bottom (0 to 7), bitboards index squares from bottom to top.
The Fix: The engine was fetching the raw matrix row. Adjusting the coordinate transformation to [7 - row] instantly brought both worlds into perfect harmony.

2. The Absolute Monarch Bug (King vs. King)

The funniest logical flaw happened during checkmate testing. The King under attack refused to lose. He would block the enemy slider's ray of attack with his own body, consider the squares behind him "safe," and happily wander around until he was literally pinned to the edge of the board. Even worse: kings could stand on adjacent squares and capture each other.

The Cause: The King was accidentally included in the blocking occupancy masks while generating specific check masks. He was effectively shading himself from the attack rays.
The Fix: Excluded the King's square from the ray-blocking lookup so that attacks correctly pass through his position during validation.

// Detect dark theme var iframe = document.getElementById('tweet-2074877950629953802-990'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2074877950629953802&theme=dark" }

3. Missing Check Restrictions (Code Refactor)

Another major flaw allowed other pieces (like knights, queens, and bishops) to make random moves even when the King was actively under check. The check_mask generation logic was breaking down when dealing with sliders.

Here is how the calculation evolved:

How it looked before:

The mask didn't differentiate piece types, leading to broken lookup tables

elif count == 1:
    self.in_check_flag = True
    checker_square = check_ers.bit_length() - 1
    checker_bit = 1 << checker_square
    check_mask = ray_from_king & ray_from_checker | checker_bit

How it looks now (Working Version):

elif count == 1:
    self.in_check_flag = True
    checker_square = check_ers.bit_length() - 1
    checker_bit = 1 << checker_square
    if checker_bit & (enemy_knight | enemy_pawns):
        check_mask = checker_bit
    else:
        # Properly determining slider type and intersection masks
        is_rook_type = ((checker_bit & enemy_rooks) or (checker_bit & enemy_queen)) and ((king_square // 8 == checker_square // 8) or (king_square % 8 == checker_square % 8))
        if is_rook_type:
            ray_from_king = self.attack_tables.get_rook_attacks(king_square, all_occ)
            ray_from_checker = self.attack_tables.get_rook_attacks(checker_square, all_occ)
        else:
            ray_from_king = self.attack_tables.get_bishop_attacks(king_square, all_occ)
            ray_from_checker = self.attack_tables.get_bishop_attacks(checker_square, all_occ)
        check_mask = ray_from_king & ray_from_checker | checker_bit

By explicitly checking if the attacking piece is a knight/pawn versus a sliding piece (rook/bishop/queen) and calculating the intersecting ray paths properly, all non-evasion moves are now correctly blocked during a check state.

Minor Fixes & Quality of Life

Pawn Promotion Crash: The matrix engine identified objects as strings ("w_pawn"), while the bitboard engine uses character notation ("P", "Q", "b"). Added a quick mapper to prevent token crashes during promotion.
Draw States: Implemented proper engine detection for Threefold Repetition and Insufficient Material (e.g., King vs King + Knight) to ensure matches can actually end legally.

What's Next? (Proof of Correctness)

Now that the visual bugs are squashed, I need to make sure the math is 100% accurate. My next milestones in Jira are setting up a Perft (Performance Test) function. It will recursively walk the move tree to a specific depth, count the total leaf nodes, and cross-reference them with official chess move datasets to catch any hidden logical edge cases.

Have you ever had a bug where your game pieces completely ignored the laws of physics or rules? Let’s talk about your favorite debugging horror stories in the comments!

python #chess #debugging #showdev

Chess Project X is Working Again... But With Some Hilarious Bugs!

Nicholas — Sat, 04 Jul 2026 13:27:18 +0000

Hey guys ;)

I’m so glad that from this milestone forward, I can finally share not just lines of code, but actual gameplay footage and visual progress. But let’s take it one step at a time.

Rewriting the Core & Fighting IDE Autocomplete

After finishing the bitboard engine backend, I duplicated my core architecture files to keep a backup. I started by rewriting the main controller file responsible for game logic and move generation.
Right away, I hit my first roadblock: the new bitboard engine lacked game state tracking variables, so I had to refactor and add state arguments directly to the BitboardEngine class.
Once that was done, I moved on to the renderer. After adapting all the UI rendering logic to read bitmasks instead of a matrix grid, I hit "Run" with high hopes.

Guess what? Instead of a beautiful chess UI, I was greeted by a wall of terminal errors. 😅
One of the funniest bugs was caused by my IDE's autocomplete. It generated move.Move() even though the Move class was located in the exact same file. I didn't even notice it until python crashed during execution.

The Invasion of the White Pawns & Regicide

After about 30 minutes of polishing micro-bugs, the Pygame window finally opened! I clicked the "Play" button, fixed a few more runtime errors, and could finally move pieces around.

However, the game state is currently a beautiful mess:

The Pawn Uprising: Due to a string formatting bug in the asset loader, every single piece on the board rendered as a white pawn. Funnily enough, they still moved according to their actual hidden piece types (e.g., a "pawn" sliding across the board like a rook).
King-on-King Violence: I forgot to properly include the king in certain restriction masks. As a result, the King could happily walk into check, stand directly next to the opposing king, and even capture the enemy king straight off the board. Talk about aggressive diplomacy.

What's Next?

I also need to tweak the promotion logic because there’s currently an issue with argument parsing between the frontend and the new backend.
On Monday, I'll begin a massive bug-hunting session to stabilize special moves (en passant, castling, promotions) and ensure total state synchronization. Once the board state is bulletproof, the environment will be fully ready for Reinforcement Learning training.
I've attached a video of this chaotic gameplay below. I hope you find it entertaining!

// Detect dark theme var iframe = document.getElementById('tweet-2073396164523626861-786'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2073396164523626861&theme=dark" }

What’s the funniest visual bug you’ve ever encountered during a major code refactor?

#python #pygame #gamedev #debugging

Ended Bitboard engine, what's next?

Nicholas — Tue, 30 Jun 2026 21:16:27 +0000

Hey guys ;)

Today I finally finished my bitboard engine! Let me take you through its architecture, and then we'll explore the next steps.

The Core Architecture

All the engine logic lives in bitboard_engine.py, which contains the Move and BitboardEngine classes. However, for everything to sync up, we rely on a few other modules: board.py (with the Board class), move_gen.py (with AttackTables), and bitboard_utils.py (housing BitboardConstants).

Here is a breakdown of what each component does:

BitboardConstants: Contains static methods for fundamental bitwise operations like setting, popping, or getting a bit (set_bit, pop_bit, get_bit), along with constant masks for the board edges (like the A/H files or ranks 1/8).
Board: Implements the state of all pieces on the board, handles the initial setup, and tracks tracking piece movements.
AttackTables: Pre-computes static attack tables for knights and kings, and builds sliding attack tables for rooks, bishops, and queens using magic numbers.
Move: Describes individual moves, storing data like piece_moved, piece_captured, and flags for special moves (is_enpassant, is_castling, is_promotion). It also handles dictionaries for correct chess notation.

The Heart of the Engine

BitboardEngine is where all these components unite. The core functionality is split into two types of functions:

Move Generation: The get_valid_moves function generates legal moves using complex bitwise masking (check masks, pin masks, enemy attack masks, and edge masks).
State Management: Functions responsible for tracking the game state (like checkmate or draws) and preparing data for the future RL model.

Honestly, this was one of the hardest parts of the project due to the sheer amount of math and theory involved, but the performance payoff compared to the matrix engine will be massive.

What's Next?

Next, I’m going to write two "bridge" functions to map bitboard indices back to the matrix format for the UI. After that, it’s time for rigorous testing. To be honest, I won't rest easy until I test it thoroughly—especially regarding special moves and check detection states.
Once the tests pass, the Reinforcement Learning phase officially begins!

Thanks for reading! If you have any questions or suggestions about magic numbers or bitboard optimization, let's chat in the comments below!

#python #chess #bitboards #showdev

My first post

Nicholas — Sat, 27 Jun 2026 19:42:20 +0000

Hey everyone, it’s my first post here ;)

Today I want to show you the project I’m currently working on. It’s Chess Project X (available as chess-project on GitHub). The development is divided into two main milestones.

Milestone 1: The Matrix Engine & UI

The first step was creating a basic chess engine from scratch along with a graphical interface. I built the initial core using matrix logic in pure Python and used the Pygame library for the UI. It includes all the neat features I personally enjoy on Chess.com: analysis arrows, highlighted squares, visual move hints, and even the native sound effects.
For the AI at this stage, I implemented a classic Minimax algorithm with Alpha-Beta pruning running at a maximum depth of 4.
Check video of gameplay on X:
// Detect dark theme var iframe = document.getElementById('tweet-2046898283264495656-673'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2046898283264495656&theme=dark" }

Milestone 2: Transitioning to Bitboards & RL

The ultimate goal is to implement a Reinforcement Learning (RL) agent. Doing this on top of a matrix-based engine would be heavily bottlenecked by performance, so I decided to rewrite the entire engine using bitboards.
Current state of the rewrite:

Implemented a new Board class.
Wrote basic bitwise utility functions (set_bit, get_bit, pop_bit).
Generated pre-computed attack masks for every piece type. For the last two days, I’ve been heavily focused on the move validation logic. There are two ways to find legal moves:
The Matrix Way: Execute make_move, check if the king is under attack, and then undo_move. It's easy to code but much slower.
The Bitboard Way: Generate all possible pin and check masks via bitwise operations. It's significantly harder to implement, but the performance payoff is massive. Right now, I've finished the logic for the king and knights. Next up: rooks, bishops, queens, and pawns. Once done, I’ll hook it up to the Pygame UI, run tests, and finally start training the RL model!

I hope you found this breakdown interesting! If you have any questions, tech suggestions, or ideas about bitboards, feel free to drop them in the comments below. See you in the next update!

#python #chess #reinforcementlearning #bitboards

DEV Community: Nicholas

Building a Chess Engine & Book Scraper - Part 4: Capping the Catalog & 4,500+ Games Cooked

"The data pipeline is perfectly stable, the automation is running smoothly, but I'm not pushing these changes to GitHub yet." 😅

♟️ Chess Database: 4,500+ Games Cooked

📖 The Book Scraper: Smart Capping & Strategy

Why cap the dataset at 200 pages? 🤔

Building a Chess Engine in Python - Part 3: Letting the DB Cook & Launching a New Book Market Scraper

"What do you do when your laptop is too weak to train a neural network and run game simulations at the same time? I'm letting the database cook, and starting a new small project."

♟️ Chess Project: The Daily Generation Quota

📖 New Project: Analyzing the Book Industry

The Pivot: From Amazon to Yakaboo

What's Next?

Building a Chess Engine in Python - Part 2: Achieving 100% Correctness & Setting Up Multicore Self-Play

It’s PEAK.

The Heartbreak (and Triumph) of Perft Testing

Multicore Self-Play & Epsilon-Greedy Exploration

Next Stop: Reinforcement Learning

Building a Chess Engine in Python - Part 1: The Immortal King & Bitboard Grid Synchronization

"I spent days writing logic masks, only to realize my Chess King became an immortal god who could capture the opposing King."

1. The Mirror Dimension ([row] vs [7 - row])

2. The Absolute Monarch Bug (King vs. King)

3. Missing Check Restrictions (Code Refactor)

How it looked before:

The mask didn't differentiate piece types, leading to broken lookup tables

python #chess #debugging #showdev

Chess Project X is Working Again... But With Some Hilarious Bugs!

Hey guys ;)

Rewriting the Core & Fighting IDE Autocomplete

The Invasion of the White Pawns & Regicide

What's Next?

Ended Bitboard engine, what's next?

Hey guys ;)

The Core Architecture

The Heart of the Engine

What's Next?

My first post

Hey everyone, it’s my first post here ;)

Milestone 1: The Matrix Engine & UI

Milestone 2: Transitioning to Bitboards & RL

1. The Mirror Dimension (`[row]` vs `[7 - row]`)