"What do you do when your laptop is too weak to train a neural network and run game simulations at the same time? I'm letting the database cook, and starting a new small project."
Hey guys ;)
Sorry for being a bit late with this update, but I promise I have some very exciting news.
Right now, my main Chess Project is entering a transition phase. To train a proper Reinforcement Learning model, I need a massive, high-quality dataset of self-play games. But running heavy simulations and coding at the same time is a heavy burden for my hardware.
So, I made a strategic decision: take a 2-week development break to generate the database, while kicking off a brand new Data Science project on the side.
Here is what's happening on both fronts!
♟️ Chess Project: The Daily Generation Quota
To keep my laptop healthy while still building a solid dataset, I’ve set up a strict automated daily schedule. From now until July 27th, I am running my parallelized self-play script with a specific daily quota:
- 716 games per day in total (distributed as 358 games per active CPU core).
- The depth distribution: 300 games at depth 2, 55 games at depth 3, and 3 games at depth 4.
Even this "limited" run takes about 2 hours and 30 minutes every single day. Once the database reaches its target size on July 27th, I will officially close the generation phase and jump straight into the model training backlog, which is already fully planned and ready in my Jira!
📖 New Project: Analyzing the Book Industry
While the chess script is running in the background, I decided to tackle a question that has been on my mind for a while: Are people actually reading fewer books nowadays?
To answer this with cold, hard data, I decided to build a custom web scraper to gather, clean, and analyze retail book market trends.
The Pivot: From Amazon to Yakaboo
Initially, my first target was Amazon. However, I quickly ran into two major roadblocks:
- Aggressive Anti-Bot Protection: Amazon does not like scrapers, making automated extraction highly inefficient without expensive proxy setups. Yep I parsed some book data from Amazon via undetected-chromedriver, but there is second problem.
- Data Limitations: The accessible, free-tier product details were too sparse for the deep analysis I wanted to conduct. To compare on Yakaboo there is a books cards with title, author, rating and count of reviews, of course price. So for my analysis I only need to get amount of pages(for example via GoogleBooks API or just right from Yakaboo details page) and interest in certain book(I'm planning to get it via Google Trends)
So, I pivoted! My new target is Yakaboo, the largest online book platform in Ukraine. It has incredibly rich metadata regarding which is exactly what I need. And of course their web source code is pretty easier to read.
I’m currently writing the automation scripts combining Selenium and BeautifulSoup to handle paginated lists and extract the raw dataset.
What's Next?
For the next two weeks:
- My CPU will be sweating to generate those 716 chess games daily.
- I will be finalizing the Yakaboo parser, cleaning the scraped book data, and preparing my first analytical dashboard.
I’m super excited to share actual data insights about what people are reading (and buying) in one of the next post!
Have you ever had to pause a heavy computational task to prevent your machine from melting? What’s your go-to tool for bypassing anti-bot systems? Let’s share some stories in the comments!
#python #datascience #webscraping #chess #showdev
Top comments (0)