"Even when your scripts are doing the heavy lifting automatically, watching them run while trying to maintain a 6/1 schedule will burn you out."
Hey guys ;)
I’m posting a day later than usual today (Thursday instead of Wednesday), and honestly, that’s part of the update. Over the past few weeks, I’ve been living in a 6 days on, 1 day off routine. When that single day off finally arrived, I felt an overwhelming pressure to cram in every missed habit, chore, and task. Instead of resting, I ended up feeling completely drained.
So, I’m changing my framework: I’m moving to a 4/3 schedule (4 days of intense focus, 3 days of genuine rest and recovery).
Here is what’s happening with my hardware, my data pipelines, and my mental bandwidth!
Memory-Efficient Data Ingestion in Pandas
On Tuesday, I officially wrapped up the main scraping phase for the Ukrainian book market. I finally ran git push on the repository and started building the processing pipeline in Pandas.
Instead of calling pd.read_json() directly, I wrote a custom loader function using with open() to stream the JSON dictionary before passing it to pd.DataFrame().
Why not just use pd.read_json()? 🤔
Standard read_json() can be a memory hog because Pandas attempts to infer structures and allocate buffer space for the entire file at once. While my raw file is currently around 50MB, this approach guarantees that as the dataset grows (after API enrichment), my machine won't choke on RAM limits.
The Case of the Missing 400 Pages
While testing the loader function and inspecting the dataset using .head(), I noticed a weird anomaly: the top entries were all children's literature (which is Category #2 in my setup).
Somewhere during the long automated execution, the first 400+ pages of the Fiction category evaporated! 😅
So here is my immediate technical checklist for today:
- [ ] Re-scrape the missing Fiction category pages.
- [ ] Run a full deduplication pass across all 16 categories.
- [ ] Integrate the Google Books API to pull missing metadata (publication years, page counts, sub-genres).
♟️ Chess RL Engine: Extending the Self-Play Deadline
Because I’m adjusting my schedule to a 4/3 split, I am officially extending my self-play generation deadline for the chess project.
People often think: "Why are you tired? You're just sitting there watching a script scrape pages or a chess engine play itself."
It’s true that the CPU does the calculation, but managing routines, tracking habits and debugging background crashes. Extending the deadline gives me the breathing room to build a high-quality dataset without turning my personal projects into a stressful second job.
Here is my screenshots of daily habits.
💬 Let's Chat!
How do you manage your schedule when building side projects? Have you ever had to restructure your work/rest ratio to prevent burnout? Let’s chat in the comments! 👇
#python #pandas #mentalhealth #datascience #showdev
Top comments (0)