DEV Community: Nicholas

A long break from post, but I’ve cocked some stuff like Walter White did.

Nicholas — Wed, 12 Aug 2026 19:11:00 +0000

Hey guys ;)
It’s been a while since my last post. Actually I was a bit busy and forgot to write about progress. And this time I really have what to say. So what happened in short?

Scraper ended data extending.
Pageviews script added pageview info for all 6k books.
My first analysis notebook is done.

So, here is a fragment of my extended data before pageviews.

And here is a fragment of dataset after pageviews.

Is pageview extending false?

As I said in a previous post, sometimes Wikipedia pageviews give some nonsense data, and it’s true. Barely it gives an exact match, more often it gives the author page, and most of the time it gives 0 or some nonsense.
When I’ll start my second analysis notebook, I’ll do something about that, maybe erase some data, for example if the saved Wiki title is in author or in title, I don’t know yet.
Here is my pageview script fragment.

First conclusions

When I began this project I asked a question, do people read less nowadays? Actually I don’t know. According to my data I couldn’t say that people read less or people read more nowadays. All I could say it’s that people still read some books indeed.
Again all ,my conclusion here will be true according to my scrapped data, custom standard of books popularity and validity for Ukraine reading culture.
So, people prefer physical books more, the most popular author is Hiro Mashima, because of the number of his manga volumes. The most popular publishers are those who produce manga and comics, again because of the number of volumes and parts. The amount of books released in the last 10 years is bigger than the amount of books released 10-20+ years ago. And the most popular categories are manga, comics, autobiography and memoirs, self-improvement and children literature. Charts from notebook I’ll add too.

What do I mean by using the words popular or famous? So in my dataset there are 6k books. And for each analysis I’ve used count. So the most popular author is who has the most amount of books. And I should remind you, that it isn’t some 6k books, it's the best books according to my custom standard.

Next step

I think the next step is going to be the last. I don’t know, maybe I’ll do all the code tomorrow, or next week. So what do I need to do?

Analyse books, their popularity according to standard and to pageviews.
Write something between code.
Write final conclusion and README.md (with some screenshots to show what this project is about very quickly, without opening notebooks).
Push and commit everything.

What do marketing tags, broken author arrays, and an unexpected email from Derek Sivers have in common?

Nicholas — Sat, 01 Aug 2026 13:38:41 +0000

Hey guys 👋

I don't have a massive technical feature to showcase today, but I do have a solid batch of data pipeline updates, a bug fix story, and a pretty inspiring personal highlight.

While refining the secondary parser for the Ukrainian book market, I ran into classic data pollution issues, stripped out marketing noise, and got a reminder of why building side projects should be fun.

Here is the weekly progress report!

Fixing the "Comma-in-Category" Bug

After rewriting the extended scraper, I noticed a subtle data formatting bug during early inspection.

Certain books on retail platforms list multiple authors. My initial DOM selection logic grabbed only the primary author string. However, because of how the HTML elements were structured, the trailing author separators (commas and secondary tags) bled directly into the Category column!

Corrupted: Category = ", Fiction, Bestsellers"
Cleaned:   Category = "Fiction"

To fix this:

Updated the scraper logic to properly parse multi-author arrays.
Manually sanitized and patched 20 affected JSON objects in the clean dataset to keep data integrity intact.

Stripping "Garbage" Categories & Preprocessing

I’m currently finishing up the final data preparation script, specifically handling empty values (NaNs) and filtering non-standard categories.

Originally, I expected platform categories to function purely as literary sub-genres (e.g., Sci-Fi, Historical Fiction, Biography). However, e-commerce platforms heavily mix genre metadata with promotional marketing tags:

"Reader's Choice"
"Top Bestsellers 2023"
"Must-Read Recommendations"

These tags provide zero value when trying to analyze actual genre trends over time. My preprocessing script is currently dropping these noise categories along with empty rows to leave a pure, genre-focused analytical dataset.

An Unexpected Reply from Derek Sivers

On a non-technical note, something really cool happened this week.

I recently finished reading Anything You Want by Derek Sivers-a short, brilliant book about entrepreneurship, independence, and doing things your own way. At the end of the book, Derek invites readers to send him an email, introduce themselves, and share what they are working on.

I decided to write to him, not really expecting a reply. To my surprise, he actually responded! He shared some thoughtful feedback and even recommended a few more great books to check out.

It was a great reminder that building side projects-whether it's a Python chess engine or a market scraper-should come from a place of curiosity and freedom.

What's Next?

Finishing the final NaN drop and category mapping pass in Pandas.
Running exploratory data analysis (EDA) to generate our very first trend charts.
Starting one of Derek's recommended books during my 3-day rest period!

Have you ever reached out directly to an author or creator who inspired you? How do you handle promotional marketing tags when cleaning e-commerce data? Let's chat in the comments below! 👇

No APIs?! Analysis is almost here, but I’ve mistaken again.

Nicholas — Wed, 29 Jul 2026 14:56:14 +0000

"Why hit API request limits for 111,000 books when a single log10 formula and direct web scraping can solve the problem?"

Hey guys ;)

Looks like my focus is completely locked on data analysis right now. I can’t help it-this project is just too fun.

Over the past two or three posts, I kept promising that I would enrich my book dataset using the Google Books API and OpenLibrary. Well... forget about the APIs.

After wrestling with rate limits, poor metadata quality, and HTML structures all week, I dropped the APIs entirely and built my own data enrichment pipeline. Here is how it happened, why I deleted 1,451 scraped books today, and what’s coming next.

The Problem with Books APIs

When I started planning the data enrichment phase for my 111,000 unique scraped books, I ran into two massive brick walls:

Google Books API Limits: The free tier caps you at 1,000 requests per day. At that rate, enriching 111k books would take over three months (I know about payment account and increasing of limits).
OpenLibrary Limitations: It handles Ukrainian literature poorly without exact ISBN matches. Title and Author search combinations gave way too many false positives.

I quickly realized I didn't actually need all 111,000 books. Thousands of them had zero reviews, no ratings, and no active search interest. Processing them was just adding noise.

Filtering the Catalog and Popularity Formula

To isolate the most relevant titles, I designed a custom metric to score and rank the entire catalog:

popularity_rating = log10(amount_of_reviews + 1) * rating

Using the logarithmic scale prevents titles with hundreds of reviews from completely breaking the ranking curve while still giving heavy weight to actual reader engagement.

This formula immediately isolated ~3,000 highly active titles. I expanded the threshold to capture the top 6,000 books for the final analytical sample.

Parsing Metadata Directly

To analyze publishing trends over time, I needed the exact publication year for a specific edition, not just the general first release year of a work.

To get that accuracy from APIs, I would need to parse the book's individual URL anyway to extract its ISBN. That's when the realization hit me: If I'm already scraping the specific book pages for ISBNs, why not just scrape the missing metadata directly from Yakaboo?

So from Monday until yesterday, I wrote a dedicated secondary parser to process my top 6,000 URL list, extract extended properties, and map them into structured category buckets.

Here is my best books selection code

Here is scraper fragment

And how I deal with categories for final dataset preprocessing

The HTML Trap and deleting 1,451 Books

It wouldn't be a real project without a good mistake.

While running the extended parser, I noticed that Yakaboo's HTML structure reuses generic CSS classes across different metadata blocks while hiding or omitting unique element IDs. Because of a misplaced selector logic, my parser was extracting Author names and saving them into the Publisher field. 😅

After 1,451 books were parsed, I caught the bug, wiped the corrupted JSON file, and fixed the DOM selector.

Today, I’m re-running the script on the top 6,000 clean URLs. The most important lesson in Data Engineering: never give up, even when you have to wipe your output and start over!

Summary & What's Next

GitHub: All changes are finally committed and pushed! I'm much more cautious here than in my Chess project, pushing only when the pipeline logic is 100% verified.
Current Task: Re-running the extended parser for the top 6k books without the Author/Publisher swap bug.
Next Goal: Data cleaning, category normalization, and generating our first analytical charts to answer if people are reading fewer books over time.

Have you ever built a custom metric to downsample a massive dataset? What’s your worst HTML scraping horror story? Let’s chat in the comments! 👇

#python #datascience #webscraping #pandas #showdev

Weekends with translator for book analysis

Nicholas — Sat, 25 Jul 2026 11:29:37 +0000

Hey guys ;)

Today I’m going to tell you about some progress in books analysis and my plans for the weekend. Let’s start.

So from my previous post, I’ve been working with the author translation problem. What is it? Okay, so for extending books data I should use some API, and for that I need an ISBN that will give a certain book with 99% accuracy, or use a Title + Author combination. When a book is from a Ukrainian publisher, it will find the book with 80-90% accuracy. But what happens if the book is from a foreign publisher? Well, in fact, the accuracy will definitely be less.

Another problem was the amount of unique books that I parsed from 16 categories. That amount surprised even me: 117,000 unique books and 63,000 unique foreign authors.

So before I found out the exact amount of unique books, I wrote a script based on the deep-translator library (Google Translator). My first few requests were done without a rate limit, so my IP got blocked. After that, I added time.sleep(2). Even after adding a dictionary for saving unique authors and reducing the number of requests, the execution time was too long - like 8 hours or more.

After that, I found an interesting library called argostranslate. The main difference is that it’s an offline library, and it works entirely on laptop resources. The process of translating took only 1h 30 min to 2 hours.

(Though someone forgot about saving! For Parquet files, Python needs the pyarrow library. So I installed pyarrow and will start the translation process on Monday.)

Okay, now about some plans. Of course, for book analysis, the next step after translation is working with an API. Now I’m thinking about a few variants:

Google Books API: With an ordinary free account, I have only 1,000 requests per day. If I connect a payment account, I’ll get 50,000 requests.
Open Library API: Free, but this API doesn’t have much info about Ukrainian publishers.

So the best option could be Google Books API. But that’s for next week!

Now it's the weekend, so I can rest, play games, and watch something interesting. Speaking of watching, first I’m going to watch some Django tutorials (because in the future I’ll definitely need it) and maybe something else.

Have a great weekend!
#python, #dataanalysis, #learning, #buildinpublic

Building a Chess AI & Book Scraper - Part 5: Memory-Efficient JSON Loading & Switching to a 4/3 Work Balance

Nicholas — Thu, 23 Jul 2026 09:40:32 +0000

"Even when your scripts are doing the heavy lifting automatically, watching them run while trying to maintain a 6/1 schedule will burn you out."

Hey guys ;)

I’m posting a day later than usual today (Thursday instead of Wednesday), and honestly, that’s part of the update. Over the past few weeks, I’ve been living in a 6 days on, 1 day off routine. When that single day off finally arrived, I felt an overwhelming pressure to cram in every missed habit, chore, and task. Instead of resting, I ended up feeling completely drained.

So, I’m changing my framework: I’m moving to a 4/3 schedule (4 days of intense focus, 3 days of genuine rest and recovery).

Here is what’s happening with my hardware, my data pipelines, and my mental bandwidth!

Memory-Efficient Data Ingestion in Pandas

On Tuesday, I officially wrapped up the main scraping phase for the Ukrainian book market. I finally ran git push on the repository and started building the processing pipeline in Pandas.

Instead of calling pd.read_json() directly, I wrote a custom loader function using with open() to stream the JSON dictionary before passing it to pd.DataFrame().

Why not just use `pd.read_json()`? 🤔

Standard read_json() can be a memory hog because Pandas attempts to infer structures and allocate buffer space for the entire file at once. While my raw file is currently around 50MB, this approach guarantees that as the dataset grows (after API enrichment), my machine won't choke on RAM limits.

The Case of the Missing 400 Pages

While testing the loader function and inspecting the dataset using .head(), I noticed a weird anomaly: the top entries were all children's literature (which is Category #2 in my setup).

Somewhere during the long automated execution, the first 400+ pages of the Fiction category evaporated! 😅

So here is my immediate technical checklist for today:

[ ] Re-scrape the missing Fiction category pages.
[ ] Run a full deduplication pass across all 16 categories.
[ ] Integrate the Google Books API to pull missing metadata (publication years, page counts, sub-genres).

♟️ Chess RL Engine: Extending the Self-Play Deadline

Because I’m adjusting my schedule to a 4/3 split, I am officially extending my self-play generation deadline for the chess project.

People often think: "Why are you tired? You're just sitting there watching a script scrape pages or a chess engine play itself."

It’s true that the CPU does the calculation, but managing routines, tracking habits and debugging background crashes. Extending the deadline gives me the breathing room to build a high-quality dataset without turning my personal projects into a stressful second job.

Here is my screenshots of daily habits.

💬 Let's Chat!

How do you manage your schedule when building side projects? Have you ever had to restructure your work/rest ratio to prevent burnout? Let’s chat in the comments! 👇

#python #pandas #mentalhealth #datascience #showdev

Building a Chess Engine & Book Scraper - Part 4: Capping the Catalog & 4,500+ Games Cooked

Nicholas — Sat, 18 Jul 2026 15:45:38 +0000

"The data pipeline is perfectly stable, the automation is running smoothly, but I'm not pushing these changes to GitHub yet." 😅

Hey guys ;)

I’m a bit tired after this week's data marathon, so I'll let the numbers and the architecture do the talking today.

Over the last seven days, my workstation has been split into two universes: generating high-quality training datasets for my chess AI on a strict daily quota, and building a scalable market analysis tool for the book industry.

Here is the weekly recap, some crucial optimizations, and the roadmap ahead.

♟️ Chess Database: 4,500+ Games Cooked

The automated parallel self-play setup I described in my previous post is performing beautifully. My hardware survived the daily quota of 716 games per day.

The Weekly Total: 6 * 716 games + 260 baseline test games = 4,556 games successfully generated and saved into the local database.

The dataset is expanding exactly as planned, laying down a bulletproof foundation before I open the Jira backlog for the Reinforcement Learning model training phase.

📖 The Book Scraper: Smart Capping & Strategy

While the chess engine runs its daily simulations, I’ve been heavily focusing on my new target platform: Yakaboo (the largest book platform in Ukraine). The platform contains 16 distinct main categories.

Initially, I went deep and scraped over 410 pages for the Fiction category alone. But looking at the data, I realized I needed to optimize my traversal mechanics. I decided to introduce a strict limit: a maximum of 200 pages per category.

Why cap the dataset at 200 pages? 🤔

Online retail storefronts naturally sort their catalog by popularity, search volume, and sales velocity on the initial pages. The deeper you go into page 300+, the higher the density of out-of-print titles, placeholders, or niche products. Scraping thousands of deep pages won't help us identify active consumer trends-it just adds unnecessary noise and processing bloat to the database.

Here is a sneak peek of how the raw item structure looks right now:

    {
        "title": "Generations in the World of Work",
        "author": "Корі Сіміллер",
        "price": "2947 грн",
        "rating": "0",
        "reviews": "0",
        "status": "Фізична книга",
        "href": "https://www.yakaboo.ua/ua/generations-in-the-world-of-work.html"
    },
    {
        "title": "Paralegal Career For Dummies",
        "author": "Ліза Зіммер Хатч",
        "price": "1832 грн",
        "rating": "0",
        "reviews": "0",
        "status": "Фізична книга",
        "href": "https://www.yakaboo.ua/ua/paralegal-career-for-dummies-3295617.html"
    },
    {
        "title": "Performance Appraisals & Phrases For Dummies",
        "author": "Кен Ллойд",
        "price": "1513 грн",
        "rating": "0",
        "reviews": "0",
        "status": "Фізична книга",
        "href": "https://www.yakaboo.ua/ua/performance-appraisals-phrases-for-dummies.html"
    },
    {
        "title": "A Pluralistic Approach to Leadership: Interdisciplinary Perspectives",
        "author": "Натан В. Хартер",
        "price": "3982 грн",
        "rating": "0",
        "reviews": "0",
        "status": "Фізична книга",
        "href": "https://www.yakaboo.ua/ua/a-pluralistic-approach-to-leadership-interdisciplinary-perspectives.html"
    },

And of course some screenshots

It's a chess database fragment. Yep some games is short, and some is more then 100 moves.

I parsed Yakaboo for 3 days and already parsed almost 23MB of data.

My laptop straggling to parse data while I'm writing posts.

And what I see as a progress. Of course I could just open Chrome and watch how pages scrolling.

Building a Chess Engine in Python - Part 3: Letting the DB Cook & Launching a New Book Market Scraper

Nicholas — Thu, 16 Jul 2026 11:29:29 +0000

"What do you do when your laptop is too weak to train a neural network and run game simulations at the same time? I'm letting the database cook, and starting a new small project."

Hey guys ;)

Sorry for being a bit late with this update, but I promise I have some very exciting news.

Right now, my main Chess Project is entering a transition phase. To train a proper Reinforcement Learning model, I need a massive, high-quality dataset of self-play games. But running heavy simulations and coding at the same time is a heavy burden for my hardware.

So, I made a strategic decision: take a 2-week development break to generate the database, while kicking off a brand new Data Science project on the side.

Here is what's happening on both fronts!

♟️ Chess Project: The Daily Generation Quota

To keep my laptop healthy while still building a solid dataset, I’ve set up a strict automated daily schedule. From now until July 27th, I am running my parallelized self-play script with a specific daily quota:

716 games per day in total (distributed as 358 games per active CPU core).
The depth distribution: 300 games at depth 2, 55 games at depth 3, and 3 games at depth 4.

Even this "limited" run takes about 2 hours and 30 minutes every single day. Once the database reaches its target size on July 27th, I will officially close the generation phase and jump straight into the model training backlog, which is already fully planned and ready in my Jira!

📖 New Project: Analyzing the Book Industry

While the chess script is running in the background, I decided to tackle a question that has been on my mind for a while: Are people actually reading fewer books nowadays?

To answer this with cold, hard data, I decided to build a custom web scraper to gather, clean, and analyze retail book market trends.

The Pivot: From Amazon to Yakaboo

Initially, my first target was Amazon. However, I quickly ran into two major roadblocks:

Aggressive Anti-Bot Protection: Amazon does not like scrapers, making automated extraction highly inefficient without expensive proxy setups. Yep I parsed some book data from Amazon via undetected-chromedriver, but there is second problem.
Data Limitations: The accessible, free-tier product details were too sparse for the deep analysis I wanted to conduct. To compare on Yakaboo there is a books cards with title, author, rating and count of reviews, of course price. So for my analysis I only need to get amount of pages(for example via GoogleBooks API or just right from Yakaboo details page) and interest in certain book(I'm planning to get it via Google Trends)

So, I pivoted! My new target is Yakaboo, the largest online book platform in Ukraine. It has incredibly rich metadata regarding which is exactly what I need. And of course their web source code is pretty easier to read.

I’m currently writing the automation scripts combining Selenium and BeautifulSoup to handle paginated lists and extract the raw dataset.

What's Next?

For the next two weeks:

My CPU will be sweating to generate those 716 chess games daily.
I will be finalizing the Yakaboo parser, cleaning the scraped book data, and preparing my first analytical dashboard.

I’m super excited to share actual data insights about what people are reading (and buying) in one of the next post!

Have you ever had to pause a heavy computational task to prevent your machine from melting? What’s your go-to tool for bypassing anti-bot systems? Let’s share some stories in the comments!

#python #datascience #webscraping #chess #showdev

Building a Chess Engine in Python - Part 2: Achieving 100% Correctness & Setting Up Multicore Self-Play

Nicholas — Sat, 11 Jul 2026 18:18:27 +0000

It’s PEAK.

Hey guys ;)

Since my last post, I’ve done a massive amount of under-the-hood work. Honestly, writing the code this time felt surprisingly smooth, but the results are a huge milestone for the project: I built a performance/correctness tester, implemented a parallelized self-play system using multiple CPU cores, and hooked it up to a database.

We are officially at the doorstep of Reinforcement Learning. Let’s break down how it all came together.

The Heartbreak (and Triumph) of Perft Testing

In my previous update, I mentioned that I was moving toward Perft (Performance Test) to verify the engine's correctness. A Perft test recursively walks the move tree to a specific depth, counts all total leaf nodes, and compares that number to standard chess data.

When I ran my first test to depth 3 and 4, my numbers were lower than the official benchmarks on chessprogramming.org. I'm not going to lie - I was really sad. After days of wrestling with bitwise masks, realizing your engine is missing thousands of moves feels brutal.

But instead of giving up, I used the test to isolate the discrepancies. It turned out the missing paths weren't deep architectural flaws, but rather sneaky edge cases in pawn logic:

The En Passant Ghost: A bug where specific en passant capture rights weren't being cleared or generated correctly.
Out of Index Crash: A rare boundary issue when pawns reached the absolute edges of the board during specific masking checks.

Once those pawn bugs were squashed, I ran the numbers again. Success! The engine officially proves its 100% mathematical correctness up to depth 4 across multiple complex custom test positions.

Multicore Self-Play & Epsilon-Greedy Exploration

With a bulletproof bitboard engine, the next challenge was generating data for our future Reinforcement Learning (RL) agent. To train a model, you need games—thousands of them.

To achieve this, I implemented a Self-Play system. Here is how the architecture works right now:

Epsilon-Greedy Minimax: I modified my classic Minimax algorithm into an epsilon-greedy version.
Opening Imitation: To prevent the engine from playing the exact same game over and over, the first 4 moves of every match are completely random. This effectively imitates traditional chess openings and forces the engine to explore a massive variety of board states.
Multicore Scaling: Running games sequentially takes ages. I utilized Python's multiprocessing features to distribute the self-play simulation across multiple CPU cores simultaneously.
Database Storage: Every single generated game, position, evaluated score, and move state is now being piped and safely recorded into a local database.

Next Stop: Reinforcement Learning

The infrastructure is ready. The engine is fast, verified as correct, scales across my CPU, and logs clean training data directly to a DB.

The next station is implementing and training the actual RL models. This is where the real experiment begins.

#python #chess #performance #multiprocessing #showdev

Building a Chess Engine in Python - Part 1: The Immortal King & Bitboard Grid Synchronization

Nicholas — Wed, 08 Jul 2026 15:40:12 +0000

"I spent days writing logic masks, only to realize my Chess King became an immortal god who could capture the opposing King."

Connecting a classic 8x8 Pygame matrix UI to a new, high-performance bitboard engine felt like a great idea until the synchronization bugs started crawling out. Over the last few days, I've been hunting down some of the most hilarious and stubborn logical errors I've encountered so far.

Let's dive into what went wrong and how I fixed it.

1. The Mirror Dimension (`[row]` vs `[7 - row]`)

After building the adapter bridge, the standard pieces moved fine, but two visual elements broke entirely: move history highlights and available move hint dots. They were rendering perfectly on the wrong side of the board.

The Cause: A classic coordinate mapping issue. While matrices count rows from top to bottom (0 to 7), bitboards index squares from bottom to top.
The Fix: The engine was fetching the raw matrix row. Adjusting the coordinate transformation to [7 - row] instantly brought both worlds into perfect harmony.

2. The Absolute Monarch Bug (King vs. King)

The funniest logical flaw happened during checkmate testing. The King under attack refused to lose. He would block the enemy slider's ray of attack with his own body, consider the squares behind him "safe," and happily wander around until he was literally pinned to the edge of the board. Even worse: kings could stand on adjacent squares and capture each other.

The Cause: The King was accidentally included in the blocking occupancy masks while generating specific check masks. He was effectively shading himself from the attack rays.
The Fix: Excluded the King's square from the ray-blocking lookup so that attacks correctly pass through his position during validation.

// Detect dark theme var iframe = document.getElementById('tweet-2074877950629953802-990'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2074877950629953802&theme=dark" }

3. Missing Check Restrictions (Code Refactor)

Another major flaw allowed other pieces (like knights, queens, and bishops) to make random moves even when the King was actively under check. The check_mask generation logic was breaking down when dealing with sliders.

Here is how the calculation evolved:

How it looked before:

The mask didn't differentiate piece types, leading to broken lookup tables

elif count == 1:
    self.in_check_flag = True
    checker_square = check_ers.bit_length() - 1
    checker_bit = 1 << checker_square
    check_mask = ray_from_king & ray_from_checker | checker_bit

How it looks now (Working Version):

elif count == 1:
    self.in_check_flag = True
    checker_square = check_ers.bit_length() - 1
    checker_bit = 1 << checker_square
    if checker_bit & (enemy_knight | enemy_pawns):
        check_mask = checker_bit
    else:
        # Properly determining slider type and intersection masks
        is_rook_type = ((checker_bit & enemy_rooks) or (checker_bit & enemy_queen)) and ((king_square // 8 == checker_square // 8) or (king_square % 8 == checker_square % 8))
        if is_rook_type:
            ray_from_king = self.attack_tables.get_rook_attacks(king_square, all_occ)
            ray_from_checker = self.attack_tables.get_rook_attacks(checker_square, all_occ)
        else:
            ray_from_king = self.attack_tables.get_bishop_attacks(king_square, all_occ)
            ray_from_checker = self.attack_tables.get_bishop_attacks(checker_square, all_occ)
        check_mask = ray_from_king & ray_from_checker | checker_bit

By explicitly checking if the attacking piece is a knight/pawn versus a sliding piece (rook/bishop/queen) and calculating the intersecting ray paths properly, all non-evasion moves are now correctly blocked during a check state.

Minor Fixes & Quality of Life

Pawn Promotion Crash: The matrix engine identified objects as strings ("w_pawn"), while the bitboard engine uses character notation ("P", "Q", "b"). Added a quick mapper to prevent token crashes during promotion.
Draw States: Implemented proper engine detection for Threefold Repetition and Insufficient Material (e.g., King vs King + Knight) to ensure matches can actually end legally.

What's Next? (Proof of Correctness)

Now that the visual bugs are squashed, I need to make sure the math is 100% accurate. My next milestones in Jira are setting up a Perft (Performance Test) function. It will recursively walk the move tree to a specific depth, count the total leaf nodes, and cross-reference them with official chess move datasets to catch any hidden logical edge cases.

Have you ever had a bug where your game pieces completely ignored the laws of physics or rules? Let’s talk about your favorite debugging horror stories in the comments!

python #chess #debugging #showdev

Chess Project X is Working Again... But With Some Hilarious Bugs!

Nicholas — Sat, 04 Jul 2026 13:27:18 +0000

Hey guys ;)

I’m so glad that from this milestone forward, I can finally share not just lines of code, but actual gameplay footage and visual progress. But let’s take it one step at a time.

Rewriting the Core & Fighting IDE Autocomplete

After finishing the bitboard engine backend, I duplicated my core architecture files to keep a backup. I started by rewriting the main controller file responsible for game logic and move generation.
Right away, I hit my first roadblock: the new bitboard engine lacked game state tracking variables, so I had to refactor and add state arguments directly to the BitboardEngine class.
Once that was done, I moved on to the renderer. After adapting all the UI rendering logic to read bitmasks instead of a matrix grid, I hit "Run" with high hopes.

Guess what? Instead of a beautiful chess UI, I was greeted by a wall of terminal errors. 😅
One of the funniest bugs was caused by my IDE's autocomplete. It generated move.Move() even though the Move class was located in the exact same file. I didn't even notice it until python crashed during execution.

The Invasion of the White Pawns & Regicide

After about 30 minutes of polishing micro-bugs, the Pygame window finally opened! I clicked the "Play" button, fixed a few more runtime errors, and could finally move pieces around.

However, the game state is currently a beautiful mess:

The Pawn Uprising: Due to a string formatting bug in the asset loader, every single piece on the board rendered as a white pawn. Funnily enough, they still moved according to their actual hidden piece types (e.g., a "pawn" sliding across the board like a rook).
King-on-King Violence: I forgot to properly include the king in certain restriction masks. As a result, the King could happily walk into check, stand directly next to the opposing king, and even capture the enemy king straight off the board. Talk about aggressive diplomacy.

What's Next?

I also need to tweak the promotion logic because there’s currently an issue with argument parsing between the frontend and the new backend.
On Monday, I'll begin a massive bug-hunting session to stabilize special moves (en passant, castling, promotions) and ensure total state synchronization. Once the board state is bulletproof, the environment will be fully ready for Reinforcement Learning training.
I've attached a video of this chaotic gameplay below. I hope you find it entertaining!

// Detect dark theme var iframe = document.getElementById('tweet-2073396164523626861-786'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2073396164523626861&theme=dark" }

What’s the funniest visual bug you’ve ever encountered during a major code refactor?

#python #pygame #gamedev #debugging

Ended Bitboard engine, what's next?

Nicholas — Tue, 30 Jun 2026 21:16:27 +0000

Hey guys ;)

Today I finally finished my bitboard engine! Let me take you through its architecture, and then we'll explore the next steps.

The Core Architecture

All the engine logic lives in bitboard_engine.py, which contains the Move and BitboardEngine classes. However, for everything to sync up, we rely on a few other modules: board.py (with the Board class), move_gen.py (with AttackTables), and bitboard_utils.py (housing BitboardConstants).

Here is a breakdown of what each component does:

BitboardConstants: Contains static methods for fundamental bitwise operations like setting, popping, or getting a bit (set_bit, pop_bit, get_bit), along with constant masks for the board edges (like the A/H files or ranks 1/8).
Board: Implements the state of all pieces on the board, handles the initial setup, and tracks tracking piece movements.
AttackTables: Pre-computes static attack tables for knights and kings, and builds sliding attack tables for rooks, bishops, and queens using magic numbers.
Move: Describes individual moves, storing data like piece_moved, piece_captured, and flags for special moves (is_enpassant, is_castling, is_promotion). It also handles dictionaries for correct chess notation.

The Heart of the Engine

BitboardEngine is where all these components unite. The core functionality is split into two types of functions:

Move Generation: The get_valid_moves function generates legal moves using complex bitwise masking (check masks, pin masks, enemy attack masks, and edge masks).
State Management: Functions responsible for tracking the game state (like checkmate or draws) and preparing data for the future RL model.

Honestly, this was one of the hardest parts of the project due to the sheer amount of math and theory involved, but the performance payoff compared to the matrix engine will be massive.

What's Next?

Next, I’m going to write two "bridge" functions to map bitboard indices back to the matrix format for the UI. After that, it’s time for rigorous testing. To be honest, I won't rest easy until I test it thoroughly—especially regarding special moves and check detection states.
Once the tests pass, the Reinforcement Learning phase officially begins!

Thanks for reading! If you have any questions or suggestions about magic numbers or bitboard optimization, let's chat in the comments below!

#python #chess #bitboards #showdev

My first post

Nicholas — Sat, 27 Jun 2026 19:42:20 +0000

Hey everyone, it’s my first post here ;)

Today I want to show you the project I’m currently working on. It’s Chess Project X (available as chess-project on GitHub). The development is divided into two main milestones.

Milestone 1: The Matrix Engine & UI

The first step was creating a basic chess engine from scratch along with a graphical interface. I built the initial core using matrix logic in pure Python and used the Pygame library for the UI. It includes all the neat features I personally enjoy on Chess.com: analysis arrows, highlighted squares, visual move hints, and even the native sound effects.
For the AI at this stage, I implemented a classic Minimax algorithm with Alpha-Beta pruning running at a maximum depth of 4.
Check video of gameplay on X:
// Detect dark theme var iframe = document.getElementById('tweet-2046898283264495656-673'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2046898283264495656&theme=dark" }

Milestone 2: Transitioning to Bitboards & RL

The ultimate goal is to implement a Reinforcement Learning (RL) agent. Doing this on top of a matrix-based engine would be heavily bottlenecked by performance, so I decided to rewrite the entire engine using bitboards.
Current state of the rewrite:

Implemented a new Board class.
Wrote basic bitwise utility functions (set_bit, get_bit, pop_bit).
Generated pre-computed attack masks for every piece type. For the last two days, I’ve been heavily focused on the move validation logic. There are two ways to find legal moves:
The Matrix Way: Execute make_move, check if the king is under attack, and then undo_move. It's easy to code but much slower.
The Bitboard Way: Generate all possible pin and check masks via bitwise operations. It's significantly harder to implement, but the performance payoff is massive. Right now, I've finished the logic for the king and knights. Next up: rooks, bishops, queens, and pawns. Once done, I’ll hook it up to the Pygame UI, run tests, and finally start training the RL model!

I hope you found this breakdown interesting! If you have any questions, tech suggestions, or ideas about bitboards, feel free to drop them in the comments below. See you in the next update!

#python #chess #reinforcementlearning #bitboards

DEV Community: Nicholas

A long break from post, but I’ve cocked some stuff like Walter White did.

Is pageview extending false?

First conclusions

Next step

What do marketing tags, broken author arrays, and an unexpected email from Derek Sivers have in common?

Fixing the "Comma-in-Category" Bug

Stripping "Garbage" Categories & Preprocessing

An Unexpected Reply from Derek Sivers

What's Next?

No APIs?! Analysis is almost here, but I’ve mistaken again.

Weekends with translator for book analysis

Building a Chess AI & Book Scraper - Part 5: Memory-Efficient JSON Loading & Switching to a 4/3 Work Balance

Memory-Efficient Data Ingestion in Pandas

Why not just use pd.read_json()? 🤔

The Case of the Missing 400 Pages

♟️ Chess RL Engine: Extending the Self-Play Deadline

💬 Let's Chat!

Building a Chess Engine & Book Scraper - Part 4: Capping the Catalog & 4,500+ Games Cooked

"The data pipeline is perfectly stable, the automation is running smoothly, but I'm not pushing these changes to GitHub yet." 😅

♟️ Chess Database: 4,500+ Games Cooked

📖 The Book Scraper: Smart Capping & Strategy

Why cap the dataset at 200 pages? 🤔

Building a Chess Engine in Python - Part 3: Letting the DB Cook & Launching a New Book Market Scraper

"What do you do when your laptop is too weak to train a neural network and run game simulations at the same time? I'm letting the database cook, and starting a new small project."

♟️ Chess Project: The Daily Generation Quota

📖 New Project: Analyzing the Book Industry

The Pivot: From Amazon to Yakaboo

What's Next?

Building a Chess Engine in Python - Part 2: Achieving 100% Correctness & Setting Up Multicore Self-Play

It’s PEAK.

The Heartbreak (and Triumph) of Perft Testing

Multicore Self-Play & Epsilon-Greedy Exploration

Next Stop: Reinforcement Learning

Building a Chess Engine in Python - Part 1: The Immortal King & Bitboard Grid Synchronization

"I spent days writing logic masks, only to realize my Chess King became an immortal god who could capture the opposing King."

1. The Mirror Dimension ([row] vs [7 - row])

2. The Absolute Monarch Bug (King vs. King)

3. Missing Check Restrictions (Code Refactor)

How it looked before:

The mask didn't differentiate piece types, leading to broken lookup tables

python #chess #debugging #showdev

Chess Project X is Working Again... But With Some Hilarious Bugs!

Hey guys ;)

Rewriting the Core & Fighting IDE Autocomplete

The Invasion of the White Pawns & Regicide

What's Next?

Ended Bitboard engine, what's next?

Hey guys ;)

The Core Architecture

The Heart of the Engine

What's Next?

My first post

Hey everyone, it’s my first post here ;)

Milestone 1: The Matrix Engine & UI

Milestone 2: Transitioning to Bitboards & RL

Why not just use `pd.read_json()`? 🤔

1. The Mirror Dimension (`[row]` vs `[7 - row]`)