Building a Chess Engine & Book Scraper - Part 4: Capping the Catalog & 4,500+ Games Cooked

"The data pipeline is perfectly stable, the automation is running smoothly, but I'm not pushing these changes to GitHub yet." 😅

Hey guys ;)

I’m a bit tired after this week's data marathon, so I'll let the numbers and the architecture do the talking today.

Over the last seven days, my workstation has been split into two universes: generating high-quality training datasets for my chess AI on a strict daily quota, and building a scalable market analysis tool for the book industry.

Here is the weekly recap, some crucial optimizations, and the roadmap ahead.

♟️ Chess Database: 4,500+ Games Cooked

The automated parallel self-play setup I described in my previous post is performing beautifully. My hardware survived the daily quota of 716 games per day.

The Weekly Total: 6 * 716 games + 260 baseline test games = 4,556 games successfully generated and saved into the local database.

The dataset is expanding exactly as planned, laying down a bulletproof foundation before I open the Jira backlog for the Reinforcement Learning model training phase.

📖 The Book Scraper: Smart Capping & Strategy

While the chess engine runs its daily simulations, I’ve been heavily focusing on my new target platform: Yakaboo (the largest book platform in Ukraine). The platform contains 16 distinct main categories.

Initially, I went deep and scraped over 410 pages for the Fiction category alone. But looking at the data, I realized I needed to optimize my traversal mechanics. I decided to introduce a strict limit: a maximum of 200 pages per category.

Why cap the dataset at 200 pages? 🤔

Online retail storefronts naturally sort their catalog by popularity, search volume, and sales velocity on the initial pages. The deeper you go into page 300+, the higher the density of out-of-print titles, placeholders, or niche products. Scraping thousands of deep pages won't help us identify active consumer trends-it just adds unnecessary noise and processing bloat to the database.

Here is a sneak peek of how the raw item structure looks right now:

    {
        "title": "Generations in the World of Work",
        "author": "Корі Сіміллер",
        "price": "2947 грн",
        "rating": "0",
        "reviews": "0",
        "status": "Фізична книга",
        "href": "https://www.yakaboo.ua/ua/generations-in-the-world-of-work.html"
    },
    {
        "title": "Paralegal Career For Dummies",
        "author": "Ліза Зіммер Хатч",
        "price": "1832 грн",
        "rating": "0",
        "reviews": "0",
        "status": "Фізична книга",
        "href": "https://www.yakaboo.ua/ua/paralegal-career-for-dummies-3295617.html"
    },
    {
        "title": "Performance Appraisals & Phrases For Dummies",
        "author": "Кен Ллойд",
        "price": "1513 грн",
        "rating": "0",
        "reviews": "0",
        "status": "Фізична книга",
        "href": "https://www.yakaboo.ua/ua/performance-appraisals-phrases-for-dummies.html"
    },
    {
        "title": "A Pluralistic Approach to Leadership: Interdisciplinary Perspectives",
        "author": "Натан В. Хартер",
        "price": "3982 грн",
        "rating": "0",
        "reviews": "0",
        "status": "Фізична книга",
        "href": "https://www.yakaboo.ua/ua/a-pluralistic-approach-to-leadership-interdisciplinary-perspectives.html"
    },

And of course some screenshots

It's a chess database fragment. Yep some games is short, and some is more then 100 moves.